- Hadoop 2.x Administration Cookbook
- Gurmukh Singh
- 511字
- 2021-07-09 20:10:26
Configuring the Hadoop Gateway node
Hadoop Gateway or edge node is a node that connects to the Hadoop cluster, but does not run any of the daemons. The purpose of an edge node is to provide an access point to the cluster and prevent users from a direct connection to critical components such as Namenode or Datanode.
Another important reason for its use is the data distribution across the cluster. If a user connects to a Datanode and performs the data copy operation hadoop fs –put file /, then one copy of the file will always go to the Datanode from which the copy command was executed. This will result in an imbalance of data across the node. If we upload a file from a node that is not a Datanode, then data will be distributed evenly for all copies of data.
In this recipe, we will configure an edge node for a Hadoop cluster.
Getting ready
For the edge node, the user needs a separate Linux machine with Java installed and the user hadoop in place.
How to do it...
sshto the new node that is to be configured as Gateway node. For example, the node name could beclient1.cluster1.com.- Set up the environment variable as discussed before. This can be done by setting the
/etc/profile.d/hadoopenv.shfile. - Copy the already configured directory
hadoop-2.7.3from Namenode to this node (client1.cluster1.com). This avoids doing all the configuration for files such ascore-site.xmlandyarn-site.xml. - The edge node just needs to know about the two master nodes of Namenode and ResourceManager. It does not need any other configuration for the time being. It does not store any data locally, unlike Namenode and Datanode.
- It only needs to write temporary files and logs. In later chapters, we will see other parameters for MapReduce and performance tuning that go on this node.
- Create a symbolic link
ln –s hadoop-2.7.3 hadoopso that the commands and Hadoop configuration files are visible. - There will be no daemon started on this node. Execute a command from the edge node to make sure the user can connect to
hadoop fs –ls /. - To verify that the edge node has been set up correctly, run the simple "Pi" estimation program from the edge node:
$ yarn jar /opt/cluster/hadoop/share/hadoop/mapreduce/hadoop-example.jar Pi 3 3
How it works...
The edge node or the Gateway node connects to Namenode for all HDFS-related operation and connects to ResourceManager for submitted jobs to the cluster.
In production, there will be more than one edge node connecting to the cluster for high availability. This is can be done by using a load balancer or DNS round-robin. No user should run any local jobs on the edge nodes or use it for doing non Hadoop-related tasks.
See also
Edge node can be used to configure many additional components, such as PIG, Hive, Sqoop, rather than installing them on the main cluster nodes like Namenode, Datanode. This way it is easy to segregate the complexity and restrict access to just edge node.
- The Configuring Hive recipe