- Hadoop 2.x Administration Cookbook
- Gurmukh Singh
- 459字
- 2021-07-09 20:10:29
Quota configuration
In a multitenancy cluster, it is important to control the utilization both in terms of HDFS space, memory, and CPU utilization. In this recipe, we will be looking at how we can restrict a user or a project from using more than the allotted HDFS space.
Getting ready
Make sure that there is a running cluster, and that the user is already well versed in the recipes that we have looked at so far.
How to do it...
- Connect to Namenode and change the user to
hadoop. - Create a directory named
projectson HDFS, as shown in the following screenshot:
- By default, there is no quota configured on any directory.
- To see what options can be set on the
projectsdirectory, use the following command:$ hadoop fs -count -q /projects
- The two leftmost fields show the namespace and disk space quota, which currently is not set, as shown in the following screenshot:
- To set the namespace quota, which will define how many inodes can be allocated for this
projectsdirectory, enter the following code. Inodes is the same as what we have in Linux. It is a data structure that defines a filesystem object:$ hdfs dfsadmin -setQuota 100 /projects
- To set the disk space quota, which will define how many blocks can be allocated for this
projectsdirectory, enter the following code:$ hdfs dfsadmin -setSpaceQuota 4G /projects
- With the preceding commands, we have set a namespace quota of 100 and a disk space quota of 4 GB, as shown in the following screenshot:
- To remove the quota on a directory, the user can use the commands as shown in the following screenshot:
How it works...
In steps 1 through 9, we configured the quota to restrict any one directory from using the entire cluster space. The namespace controls how many files can be created in that path and the space quota tells us what the total size will be.
So now it is up to the user whether they want to create just a single file of 4 GB or 10 smaller files. This way, we are forcing the user not to create small files, as, if they do, they will run out of namespace quota.
The following commands show the total space available in the cluster, but the first command takes into account the replication factor:
$ hadoop fs -count -q / 9223372036854775807 9223372036854774414 $ hdfs dfsadmin -report | head Configured Capacity: 378046439424 (352.08 GB) Present Capacity: 357588590592 (333.03 GB) DFS Remaining: 355012579328 (330.63 GB) DFS Used: 2576011264 (2.40 GB)