- Hadoop Beginner's Guide
- Garry Turkington
- 266字
- 2021-07-29 16:51:40
Time for action – WordCount the easy way
Let's revisit WordCount, but this time use some of these predefined map
and reduce
implementations:
- Create a new
WordCountPredefined.java
file containing the following code:import org.apache.hadoop.conf.Configuration ; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.map.TokenCounterMapper ; import org.apache.hadoop.mapreduce.lib.reduce.IntSumReducer ; public class WordCountPredefined { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "word count1"); job.setJarByClass(WordCountPredefined.class); job.setMapperClass(TokenCounterMapper.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }
- Now compile, create the JAR file, and run it as before.
- Don't forget to delete the output directory before running the job, if you want to use the same location. Use the
hadoop fs -rmr
output, for example.
What just happened?
Given the ubiquity of WordCount as an example in the MapReduce world, it's perhaps not entirely surprising that there are predefined Mapper
and Reducer
implementations that together realize the entire WordCount solution. The TokenCounterMapper
class simply breaks each input line into a series of (token, 1)
pairs and the IntSumReducer
class provides a final count by summing the number of values for each key.
There are two important things to appreciate here:
- Though WordCount was doubtless an inspiration for these implementations, they are in no way specific to it and can be widely applicable
- This model of having reusable mapper and reducer implementations is one thing to remember, especially in combination with the fact that often the best starting point for a new MapReduce job implementation is an existing one
推薦閱讀
- Microsoft Dynamics CRM Customization Essentials
- 精通MATLAB神經(jīng)網(wǎng)絡(luò)
- LabVIEW虛擬儀器從入門到測控應(yīng)用130例
- 機(jī)器學(xué)習(xí)及應(yīng)用(在線實驗+在線自測)
- 輕松學(xué)PHP
- 計算機(jī)應(yīng)用基礎(chǔ)·基礎(chǔ)模塊
- Mobile DevOps
- 計算機(jī)控制技術(shù)
- Learning Social Media Analytics with R
- STM32G4入門與電機(jī)控制實戰(zhàn):基于X-CUBE-MCSDK的無刷直流電機(jī)與永磁同步電機(jī)控制實現(xiàn)
- 從零開始學(xué)C++
- Building a BeagleBone Black Super Cluster
- 網(wǎng)絡(luò)脆弱性掃描產(chǎn)品原理及應(yīng)用
- 空間機(jī)器人
- MPC5554/5553微處理器揭秘