- Hadoop MapReduce v2 Cookbook（Second Edition）
- Thilina Gunarathne
- 360字
- 2021-07-23 20:32:54
Saving money using Amazon EC2 Spot Instances to execute EMR job flows
Amazon EC2 Spot Instances allow us to purchase underutilized EC2 compute resources at a significant discount. The prices of Spot Instances change depending on the demand. We can submit bids for the Spot Instances and we receive the requested compute instances if our bid exceeds the current Spot Instance price. Amazon bills these instances based on the actual Spot Instance price, which can be lower than your bid. Amazon will terminate your instances, if the Spot Instance price exceeds your bid. However, Amazon does not charge for partial Spot Instance hours if Amazon terminated your instances. You can find more information on Amazon EC2 Spot Instances at http://aws.amazon.com/ec2/spot-instances/.
Amazon EMR supports using Spot Instances both as master as well as worker compute instances. Spot Instances are ideal to execute nontime critical computations such as batch jobs.
How to do it...
The following steps show you how to use Amazon EC2 Spot Instances with Amazon Elastic MapReduce to execute the WordCount MapReduce application:
- Follow steps 1 to 9 of the Running Hadoop MapReduce v2 computations using Amazon Elastic MapReduce recipe.
- Configure your EMR cluster to use Spot Instances in the Hardware Configuration section. (Refer to step 10 of the Running Hadoop MapReduce v2 computations using Amazon Elastic MapReduce recipe). In the Hardware Configuration section, select the Request Spot checkboxes next to the instance types.
- Specify your bid price in the Bid price textboxes. You can find the Spot Instance pricing history in the Spot Requests window of the Amazon EC2 console (https://console.aws.amazon.com/ec2).
- Follow steps 11 to 16 of the Running Hadoop MapReduce v2 computations using Amazon Elastic MapReduce recipe.
There's more...
You can also run the EMR computations on a combination of traditional EC2 on-demand instances and EC2 Spot Instances, safe guarding your computation against possible Spot Instance terminations.
Since Amazon bills the Spot Instances using the current Spot price irrespective of your bid price, it is a good practice to not set the Spot Instance price too low to avoid the risk of frequent terminations.
See also
The Running Hadoop MapReduce v2 computations using Amazon Elastic MapReduce recipe.