You can use the Hadoop Distributed File System (HDFS) Balancer to analyze the distribution of data blocks and redistribute data that is stored on DataNodes. This topic describes how to use the HDFS Balancer and optimize the performance of the HDFS Balancer.
Use the HDFS Balancer
Method 1: Run the hdfs balancer command
hdfs balancer
[-threshold <threshold>]
[-policy <policy>]
[-exclude [-f <hosts-file> | <comma-separated list of hosts>]]
[-include [-f <hosts-file> | <comma-separated list of hosts>]]
[-source [-f <hosts-file> | <comma-separated list of hosts>]]
[-blockpools <comma-separated list of blockpool ids>]
[-idleiterations <idleiterations>]
Parameter | Description |
---|---|
threshold | The threshold of the disk usage, in percentage. Default value: 10%. This value ensures that the disk usage on each DataNode differs from the overall usage in the cluster by no more than 10%. If the overall usage of the cluster is high, set this parameter to a smaller value. If a large number of new nodes are added to the cluster, you can set this parameter to a larger value to move data from the high-usage nodes to the low-usage nodes. |
policy | The balancing policy. Valid values:
|
exclude | Excludes specified DataNodes. |
include | Specifies DataNodes on which the balancing operation is performed. |
source | The DataNode that serves as the source node. |
blockpools | The block pools in which the HDFS Balancer runs. |
idleiterations | The maximum number of idle loops that are allowed. Default value: 5. |
Method 2: Run the start-balancer.sh tool
The operation of running the start-balancer.sh tool is equivalent to the operation of running the hdfs daemon start balancer command. To use this tool, perform the following steps:
- Log on to a node of the cluster to be configured. For more information, see Log on to a cluster.
- Optional:Run the following command to change the maximum bandwidth of the HDFS Balancer:
hdfs dfsadmin -setBalancerBandwidth <bandwidth in bytes per second>
Note<bandwidth in bytes per second>
specifies the maximum bandwidth. For example, if you want to set the maximum bandwidth to 20 MB/s, set the value of <bandwidth in bytes per second> to 20971520. The complete command ishdfs dfsadmin -setBalancerBandwidth 20971520
. If the cluster is heavily loaded, we recommend that you set the value to 209715200, which indicates that the maximum bandwidth is 200 MB/s. If the cluster is idle, we recommend that you set the value to 1073741824, which indicates that the maximum bandwidth is 1 GB/s. - Run the following commands to switch to the hdfs user and run the HDFS Balancer:
- Data lake cluster
su hdfs /opt/apps/HDFS/hdfs-current/sbin/start-balancer.sh -threshold 5
- Hadoop cluster
su hdfs /usr/lib/hadoop-current/sbin/start-balancer.sh -threshold 5
- Data lake cluster
- Run the following command to go to the installation directory of HDFS:
- Data lake cluster
cd /var/log/emr/hadoop/
- Hadoop cluster
cd /var/log/hadoop-hdfs
- Data lake cluster
- Run the following command to check the status of the HDFS Balancer:
- Data lake cluster
tailf hadoop-hdfs-balancer-master-1-1.c-xxx.log
- Hadoop cluster
tailf hadoop-hdfs-balancer-emr-header-1.cluster-xxx.log
Note In the preceding commands,hadoop-hdfs-balancer-master-1-1.c-xxx.log
orhadoop-hdfs-balancer-emr-header-xx.cluster-xxx.log
is the name of the log file that is obtained in Step 4.
If the command output includes
Successfully
, the HDFS Balancer is running. - Data lake cluster
Parameters used to optimize the performance of the HDFS Balancer
- Client configurations
Parameter Description dfs.balancer.dispatcherThreads The number of dispatcher threads used by the HDFS Balancer to determine the blocks that need to be moved. Before the HDFS Balancer moves a specific amount of data between two DataNodes, the HDFS Balancer repeatedly retrieves block lists for moving blocks until the required amount of data is scheduled. Note Default value: 200.dfs.balancer.rpc.per.sec The number of remote procedure calls (RPCs) sent by dispatcher threads per second. Default value: 20. Before the HDFS Balancer moves data between two DataNodes, the HDFS Balancer uses dispatcher threads to repeatedly send the getBlocks() RPC to the NameNode. This results in a heavy load on the NameNode. To prevent this issue and balance the cluster load, we recommend that you set this parameter to limit the number of RPCs sent per second.
For example, you can decrease the value of the parameter by 10 or 5 for a cluster with a high load to minimize the impact on the overall moving progress.
dfs.balancer.getBlocks.size The total data size of the blocks moved each time. Before the HDFS Balancer moves data between two DataNodes, the HDFS Balancer repeatedly retrieves block lists for moving blocks until the required amount of data is scheduled. By default, the size of blocks in each block list is 2 GB. When the NameNode receives a getBlocks() RPC, the NameNode is locked. If an RPC queries a large number of blocks, the NameNode is locked for a long period of time. This slows down data writing. To prevent this issue, we recommend that you set this parameter based on the NameNode load. dfs.balancer.moverThreads Default value: 1000. The total number of threads that are used to move blocks. Each block move requires a thread.
- DataNode configurations
Parameter Description dfs.datanode.balance.bandwidthPerSec The bandwidth for each DataNode to balance the workloads of the cluster. We recommend that you set the bandwidth to 100 MB/s. You can also set the dfsadmin -setBalancerBandwidth parameter to adjust the bandwidth. You do not need to restart DataNodes.
For example, you can increase the bandwidth when the cluster load is low and decrease the bandwidth when the cluster load is high.
dfs.datanode.balance.max.concurrent.moves Default value: 5.
The maximum number of concurrent block moves that are allowed in a DataNode. Set this parameter based on the number of disks. We recommend that you set this parameter to
4 × Number of disks
as the upper limit for a DataNode.For example, if a DataNode has 28 disks, set this parameter to 28 on the HDFS Balancer and 112 on the DataNode. Adjust the value based on the cluster load. Increase the value when the cluster load is low and decrease the value when the cluster load is high.
FAQ
Q: Why is the difference approximate to 20% after the balancing even though the threshold parameter is set to 10%?
A: The threshold parameter is used to keep the usage of each DataNode not higher or lower than the average usage of the cluster. Therefore, the difference between the DataNodes with the highest and the lowest usage may be 20% after the balancing. To reduce the difference, you can set the threshold parameter to 5%.