- Hadoop MapReduce v2 Cookbook（Second Edition）
- Thilina Gunarathne
- 460字
- 2021-07-23 20:32:56
Using EMR bootstrap actions to configure VMs for the Amazon EMR jobs
EMR bootstrap actions provide us a mechanism to configure the EC2 instances before running our MapReduce computations. Examples of bootstrap actions include providing custom configurations for Hadoop, installing any dependent software, distributing a common dataset, and so on. Amazon provides a set of predefined bootstrap actions as well as allowing us to write our own custom bootstrap actions. EMR runs the bootstrap actions in each instance before Hadoop cluster services are started.
In this recipe, we are going to use a stop words list to filter out the common words from our WordCount sample. We download the stop words list to the workers using a custom bootstrap action.
How to do it...
The following steps show you how to download a file to all the EC2 instances of an EMR computation using a bootstrap script:
- Save the following script to a file named
download-stopwords.sh. Upload the file to a Blob container in the Amazon S3. This custom bootstrap file downloads a stop words list to each instance and copies it to a pre-designated directory inside the instance.#!/bin/bash set -e wget http://www.textfixer.com/resources/common-english-words-with-contractions.txt mkdir –p /home/hadoop/stopwords mv common-english-words-with-contractions.txt /home/hadoop/stopwords
- Complete steps 1 to 10 of the Running Hadoop MapReduce computations using Amazon Elastic MapReduce recipe in this chapter.
- Select the Add Bootstrap Actions option in the Bootstrap Actions tab. Select Custom Action in the Add Bootstrap Actions drop-down box. Click on Configure and add. Give a name to your action in the Name textbox and provide the S3 path of the location where you uploaded the
download-stopwords.shfile in the S3 location textbox. Click on Add.
- Add Steps if needed.
- Click on the Create Cluster button to launch instances and to deploy the MapReduce cluster.
- Click on Refresh in the EMR console and go to your Cluster Details page to view the details of the cluster.
There's more...
Amazon provides us with the following predefined bootstrap actions:
configure-daemons: This allows us to set Java Virtual Machine (JVM) options for the Hadoop daemons, such as the heap size and garbage collection behavior.configure-hadoop: This allows us to modify the Hadoop configuration settings. Either we can upload a Hadoop configuration XML or we can specify individual configuration options as key-value pairs.memory-intensive: This allows us to configure the Hadoop cluster for memory-intensive workloads.run-if: This allows us to run bootstrap actions based on a property of an instance. This action can be used in scenarios where we want to run a command only in the Hadoop master node.
You can also create shutdown actions by writing scripts to a designated directory in the instance. Shutdown actions are executed after the job flow is terminated.
Refer to http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/Bootstrap.html for more information.
- 微信公眾平臺與小程序開發(fā)：從零搭建整套系統(tǒng)
- Redis Applied Design Patterns
- Raspberry Pi Networking Cookbook（Second Edition）
- Learning Informatica PowerCenter 10.x（Second Edition）
- PySide GUI Application Development（Second Edition）
- 全棧自動化測試實戰(zhàn)：基于TestNG、HttpClient、Selenium和Appium
- Android程序設(shè)計基礎(chǔ)
- Learning Apache Cassandra
- Building Dynamics CRM 2015 Dashboards with Power BI
- GitHub入門與實踐
- Android應(yīng)用開發(fā)攻略
- Microsoft Windows Identity Foundation Cookbook
- jQuery Essentials
- Test-Driven Java Development（Second Edition）
- 開源心法