- Hadoop MapReduce v2 Cookbook（Second Edition）
- Thilina Gunarathne
- 460字
- 2021-07-23 20:32:56
Using EMR bootstrap actions to configure VMs for the Amazon EMR jobs
EMR bootstrap actions provide us a mechanism to configure the EC2 instances before running our MapReduce computations. Examples of bootstrap actions include providing custom configurations for Hadoop, installing any dependent software, distributing a common dataset, and so on. Amazon provides a set of predefined bootstrap actions as well as allowing us to write our own custom bootstrap actions. EMR runs the bootstrap actions in each instance before Hadoop cluster services are started.
In this recipe, we are going to use a stop words list to filter out the common words from our WordCount sample. We download the stop words list to the workers using a custom bootstrap action.
How to do it...
The following steps show you how to download a file to all the EC2 instances of an EMR computation using a bootstrap script:
- Save the following script to a file named
download-stopwords.sh. Upload the file to a Blob container in the Amazon S3. This custom bootstrap file downloads a stop words list to each instance and copies it to a pre-designated directory inside the instance.#!/bin/bash set -e wget http://www.textfixer.com/resources/common-english-words-with-contractions.txt mkdir –p /home/hadoop/stopwords mv common-english-words-with-contractions.txt /home/hadoop/stopwords
- Complete steps 1 to 10 of the Running Hadoop MapReduce computations using Amazon Elastic MapReduce recipe in this chapter.
- Select the Add Bootstrap Actions option in the Bootstrap Actions tab. Select Custom Action in the Add Bootstrap Actions drop-down box. Click on Configure and add. Give a name to your action in the Name textbox and provide the S3 path of the location where you uploaded the
download-stopwords.shfile in the S3 location textbox. Click on Add.
- Add Steps if needed.
- Click on the Create Cluster button to launch instances and to deploy the MapReduce cluster.
- Click on Refresh in the EMR console and go to your Cluster Details page to view the details of the cluster.
There's more...
Amazon provides us with the following predefined bootstrap actions:
configure-daemons: This allows us to set Java Virtual Machine (JVM) options for the Hadoop daemons, such as the heap size and garbage collection behavior.configure-hadoop: This allows us to modify the Hadoop configuration settings. Either we can upload a Hadoop configuration XML or we can specify individual configuration options as key-value pairs.memory-intensive: This allows us to configure the Hadoop cluster for memory-intensive workloads.run-if: This allows us to run bootstrap actions based on a property of an instance. This action can be used in scenarios where we want to run a command only in the Hadoop master node.
You can also create shutdown actions by writing scripts to a designated directory in the instance. Shutdown actions are executed after the job flow is terminated.
Refer to http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/Bootstrap.html for more information.
- 軟件界面交互設計基礎
- Django：Web Development with Python
- Node.js Design Patterns
- Mastering JavaScript High Performance
- 小型編譯器設計實踐
- 大學計算機基礎
- BeagleBone Robotic Projects（Second Edition）
- 區塊鏈架構之美：從比特幣、以太坊、超級賬本看區塊鏈架構設計
- MongoDB Cookbook（Second Edition）
- Groovy 2 Cookbook
- JSP大學實用教程
- 深度學習的數學：使用Python語言
- 前端Serverless：面向全棧的無服務器架構實戰
- 網絡工程方案設計與實施（第二版）
- CorelDRAW X6中文版應用教程（第二版）