- Hadoop 2.x Administration Cookbook
- Gurmukh Singh
- 500字
- 2021-07-09 20:10:27
Decommissioning nodes
There will always be failures in clusters, such as hardware issues or a need to upgrade nodes. This should be done in a graceful manner, without any data loss.
When the Datanode daemon is stopped on a Datanode, it takes approximately ten minutes for the Namenode to remove that node. This has to do with the heartbeat retry interval. At any time, we can abruptly remove the Datanode, but it can result in data loss.
It is recommended that you opt for the graceful removal of the node from the cluster, as this ensures that all the data on that node is drained.
Getting ready
For the following steps, we assume that the cluster that is up and running with Datanodes is in a healthy state and the one with the Datanode dn1.cluster1.com needs maintenance and must be removed from the cluster. We will login to the Namenode and make changes there.
How to do it...
- ssh to Namenode and edit the file
hdfs-site.xmlby adding the following property to it:<property> <name>dfs.hosts.exclude</name> <value>/home/hadoop/excludes</value> <final>true</final> </property>
- Make sure the file
excludesis readable by the userhadoop. - Restart the Namenode daemon for the property to take effect:
$ hadoop-daemons.sh stop namenode $ hadoop-daemons.sh start namenode
- A restart of Namenode is required only when any property is changed in the file. Once the property is in place, Namenode can read the changes to the contents of the file
excludesby simply refreshing nodes. - Add the
dn1.cluster1.comnode to the fileexcludes:$ cat excludes dn1.cluster1.com
- After adding the node to the file, we just need to reload the file by doing the following:
$ hadoop dfsadmin -refreshNodes - After sometime, the node will be decommissioned. The time will vary according to the data the particular Datanode had. We can see the decommissioned nodes using the following:
$ hdfs dfsadmin -report - The preceding command will list the nodes in the cluster, and against the
dn1.cluster1.comnode we can see that its status will either be decommissioning or decommissioned.
How it works...
Let's have a look at what we did throughout this recipe.
In steps 1 through 6, we added the new property to the hdfs-site.xml file and then restarted Namenode to make it aware of the changes. Once the property is in place, the Namenode is aware of the excludes file, and it can be asked to re-read by simply refreshing the node list, as done in step 6.
With these steps, the data on the Datanode dn1.cluster1.com will be moved to other nodes in the cluster, and once the data has been drained, the Datanode daemon on the node will be shutdown. During the process, the node will change the status from normal to decommissioning and then to decommissioned.
Care must be taken while decommissioning nodes in the cluster. The user should not decommission multiple nodes at a time as this will generate lot of network traffic and cause congestion and data loss.
See also
- The Add nodes to the cluster recipe
- 機器學習實戰：基于Sophon平臺的機器學習理論與實踐
- 亮劍.NET：.NET深入體驗與實戰精要
- Mastering Mesos
- Getting Started with Clickteam Fusion
- Blockchain Quick Start Guide
- 條碼技術及應用
- 精通特征工程
- Photoshop CS3圖像處理融會貫通
- Implementing Oracle API Platform Cloud Service
- Chef：Powerful Infrastructure Automation
- 中國戰略性新興產業研究與發展·智能制造裝備
- 走近大數據
- 深度學習與目標檢測
- Cortex-M3嵌入式處理器原理與應用
- 數據庫基礎：Access