- Hadoop MapReduce v2 Cookbook（Second Edition）
- Thilina Gunarathne
- 224字
- 2021-07-23 20:32:53
Running the WordCount program in a distributed cluster environment
This recipe describes how to run a MapReduce computation in a distributed Hadoop v2 cluster.
Getting ready
Start the Hadoop cluster by following the Setting up HDFS recipe or the Setting up Hadoop ecosystem in a distributed cluster environment using a Hadoop distribution recipe.
How to do it...
Now let's run the WordCount sample in the distributed Hadoop v2 setup:
- Upload the
wc-inputdirectory in the source repository to the HDFS filesystem. Alternatively, you can upload any other set of text documents as well.$ hdfs dfs -copyFromLocal wc-input . - Execute the WordCount example from the
HADOOP_HOMEdirectory:$ hadoop jar hcb-c1-samples.jar \ chapter1.WordCount \ wc-input wc-output
- Run the following commands to list the output directory and then look at the results:
$hdfs dfs -ls wc-output Found 3 items -rw-r--r-- 1 joesupergroup0 2013-11-09 09:04 /data/output1/_SUCCESS drwxr-xr-x - joesupergroup0 2013-11-09 09:04 /data/output1/_logs -rw-r--r-- 1 joesupergroup1306 2013-11-09 09:04 /data/output1/part-r-00000 $ hdfs dfs -cat wc-output/part*
How it works...
When we submit a job, YARN would schedule a MapReduce ApplicationMaster to coordinate and execute the computation. ApplicationMaster requests the necessary resources from the ResourceManager and executes the MapReduce computation using the containers it received from the resource request.
There's more...
You can also see the results of the WordCount application through the HDFS monitoring UI by visiting http://NAMANODE:50070.
推薦閱讀
- Extending Jenkins
- Python 3.7網絡爬蟲快速入門
- Maven Build Customization
- ASP.NET動態網頁設計教程（第三版）
- Animate CC二維動畫設計與制作（微課版）
- Functional Programming in JavaScript
- C語言程序設計
- Visual Basic程序設計與應用實踐教程
- PHP+MySQL網站開發項目式教程
- QGIS：Becoming a GIS Power User
- Windows Server 2016 Automation with PowerShell Cookbook（Second Edition）
- Learning DHTMLX Suite UI
- 小程序，巧應用：微信小程序開發實戰（第2版）
- Python+Tableau數據可視化之美
- Web前端應用開發技術