官术网_书友最值得收藏!

Time for action – WordCount the easy way

Let's revisit WordCount, but this time use some of these predefined map and reduce implementations:

  1. Create a new WordCountPredefined.java file containing the following code:
    import org.apache.hadoop.conf.Configuration ;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    import org.apache.hadoop.mapreduce.lib.map.TokenCounterMapper ;
    import org.apache.hadoop.mapreduce.lib.reduce.IntSumReducer ;
    
    public class WordCountPredefined
    {   
        public static void main(String[] args) throws Exception
        {
            Configuration conf = new Configuration();
            Job job = new Job(conf, "word count1");
            job.setJarByClass(WordCountPredefined.class);
            job.setMapperClass(TokenCounterMapper.class);
            job.setReducerClass(IntSumReducer.class);
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(IntWritable.class);
            FileInputFormat.addInputPath(job, new Path(args[0]));
            FileOutputFormat.setOutputPath(job, new Path(args[1]));
            System.exit(job.waitForCompletion(true) ? 0 : 1);
        }
    }
  2. Now compile, create the JAR file, and run it as before.
  3. Don't forget to delete the output directory before running the job, if you want to use the same location. Use the hadoop fs -rmr output, for example.

What just happened?

Given the ubiquity of WordCount as an example in the MapReduce world, it's perhaps not entirely surprising that there are predefined Mapper and Reducer implementations that together realize the entire WordCount solution. The TokenCounterMapper class simply breaks each input line into a series of (token, 1) pairs and the IntSumReducer class provides a final count by summing the number of values for each key.

There are two important things to appreciate here:

  • Though WordCount was doubtless an inspiration for these implementations, they are in no way specific to it and can be widely applicable
  • This model of having reusable mapper and reducer implementations is one thing to remember, especially in combination with the fact that often the best starting point for a new MapReduce job implementation is an existing one
主站蜘蛛池模板: 洛川县| 昌都县| 射阳县| 开江县| 鄂温| 手游| 九江市| 怀柔区| 桐庐县| 顺义区| 赣州市| 西乡县| 阿鲁科尔沁旗| 合川市| 高邮市| 建始县| 全椒县| 凤山县| 永仁县| 宜兰县| 鄄城县| 徐闻县| 乡城县| 锦屏县| 喜德县| 温宿县| 石泉县| 峨山| 和政县| 时尚| 淳安县| 鄂温| 共和县| 昌邑市| 鸡西市| 泗洪县| 寻乌县| 贵阳市| 正蓝旗| 当雄县| 宽甸|