- Hadoop 2.x Administration Cookbook
- Gurmukh Singh
- 459字
- 2021-07-09 20:10:29
Quota configuration
In a multitenancy cluster, it is important to control the utilization both in terms of HDFS space, memory, and CPU utilization. In this recipe, we will be looking at how we can restrict a user or a project from using more than the allotted HDFS space.
Getting ready
Make sure that there is a running cluster, and that the user is already well versed in the recipes that we have looked at so far.
How to do it...
- Connect to Namenode and change the user to
hadoop. - Create a directory named
projectson HDFS, as shown in the following screenshot:
- By default, there is no quota configured on any directory.
- To see what options can be set on the
projectsdirectory, use the following command:$ hadoop fs -count -q /projects
- The two leftmost fields show the namespace and disk space quota, which currently is not set, as shown in the following screenshot:
- To set the namespace quota, which will define how many inodes can be allocated for this
projectsdirectory, enter the following code. Inodes is the same as what we have in Linux. It is a data structure that defines a filesystem object:$ hdfs dfsadmin -setQuota 100 /projects
- To set the disk space quota, which will define how many blocks can be allocated for this
projectsdirectory, enter the following code:$ hdfs dfsadmin -setSpaceQuota 4G /projects
- With the preceding commands, we have set a namespace quota of 100 and a disk space quota of 4 GB, as shown in the following screenshot:
- To remove the quota on a directory, the user can use the commands as shown in the following screenshot:
How it works...
In steps 1 through 9, we configured the quota to restrict any one directory from using the entire cluster space. The namespace controls how many files can be created in that path and the space quota tells us what the total size will be.
So now it is up to the user whether they want to create just a single file of 4 GB or 10 smaller files. This way, we are forcing the user not to create small files, as, if they do, they will run out of namespace quota.
The following commands show the total space available in the cluster, but the first command takes into account the replication factor:
$ hadoop fs -count -q / 9223372036854775807 9223372036854774414 $ hdfs dfsadmin -report | head Configured Capacity: 378046439424 (352.08 GB) Present Capacity: 357588590592 (333.03 GB) DFS Remaining: 355012579328 (330.63 GB) DFS Used: 2576011264 (2.40 GB)
- 大學計算機信息技術導論
- 精通MATLAB圖像處理
- 計算機原理
- Implementing AWS：Design，Build，and Manage your Infrastructure
- Mastering ServiceNow Scripting
- 網站前臺設計綜合實訓
- 人工智能技術入門
- Silverlight 2完美征程
- Machine Learning with Spark（Second Edition）
- 貫通Java Web輕量級應用開發
- 計算機辦公應用培訓教程
- Keras Reinforcement Learning Projects
- 大話數據科學：大數據與機器學習實戰（基于R語言）
- 微計算機原理及應用
- SQL語言與數據庫操作技術大全