- Hadoop 2.x Administration Cookbook
- Gurmukh Singh
- 392字
- 2021-07-09 20:10:29
HDFS health and FSCK
The health of the filesystem is very important for data retrieval and optimal performance. In a distributed system, it becomes more critical to maintain the good health of the HDFS filesystem so as to ensure block replication and near-parallel streaming of data blocks.
In this recipe, we will see how to check the health of the filesystem and do repairs, if any are needed.
Getting ready
Make sure you have a running cluster that has already been up for a few days with data. We can run the commands on a new cluster as well, but for the sake of this lab, it will give you more insights if it is run on a cluster with a large dataset.
How to do it...
- ssh to the
master1.cyrus.comNamenode and change the user tohadoop. - To check the HDFS root filesystem, execute the
hdfs fsck /command, as shown in the following screenshot:
- We can also check the status of just one file instead of the entire filesystem, as shown in the following screenshot:
- The output of the
fsckcommand will show the blocks for a file, the replication status, whether blocks are corrupted, and many more details, as shown in the following screenshot:
- We can also look at how the blocks of a file are laid across the cluster using the commands as shown in the following screenshot:
- In the cluster named
cyrus, you can see that there are some corrupt blocks. We can simulate this by manually deleting a block of a file on the lower filesystem. Each of the HDFS blocks, is a file at the lower filesystem such as EXT4.
- The corrupt blocks can be fixed by deleting them, and for an under replicated block we can use the
hdfs dfs -setrep 2 /input/new.txtcommand, so that a particular file is set to the desired number of replications. If we need to set many files to a specified number of replications, just loop through the list and do asetrepon them.
How it works...
The hdfs fsck /command is similar to the Linux fsck command. In Hadoop, it does not repair the filesystem automatically and needs a manual intervention. To see what options there are for this command, please use the hdfs fsck –help help command.
See also
- The Configuring rack awareness recipe
- Internet接入·網(wǎng)絡(luò)安全
- 輕松學(xué)C語言
- Canvas LMS Course Design
- VB語言程序設(shè)計
- Visual C++編程全能詞典
- 電氣控制與PLC技術(shù)應(yīng)用
- 傳感器與新聞
- R Machine Learning Projects
- ESP8266 Robotics Projects
- PLC與變頻技術(shù)應(yīng)用
- 中文版AutoCAD 2013高手速成
- 網(wǎng)絡(luò)脆弱性掃描產(chǎn)品原理及應(yīng)用
- 步步驚“芯”
- Access 2007數(shù)據(jù)庫入門與實例應(yīng)用金典
- JSP通用范例開發(fā)金典