You can use the Hadoop Distributed File System (HDFS) Balancer to analyze the distribution of data blocks and redistribute data that is stored on DataNodes. This topic describes how to use the HDFS Balancer and optimize the performance of the HDFS Balancer.

Use the HDFS Balancer

Method 1: Run the hdfs balancer command

Run the following command to configure the HDFS Balancer:
hdfs balancer
[-threshold <threshold>]
[-policy <policy>]
[-exclude [-f <hosts-file> | <comma-separated list of hosts>]]
[-include [-f <hosts-file> | <comma-separated list of hosts>]]
[-source [-f <hosts-file> | <comma-separated list of hosts>]]
[-blockpools <comma-separated list of blockpool ids>]
[-idleiterations <idleiterations>]
The following table describes the parameters of the HDFS Balancer.
ParameterDescription
thresholdThe threshold of the disk usage, in percentage.

Default value: 10%. This value ensures that the disk usage on each DataNode differs from the overall usage in the cluster by no more than 10%.

If the overall usage of the cluster is high, set this parameter to a smaller value.

If a large number of new nodes are added to the cluster, you can set this parameter to a larger value to move data from the high-usage nodes to the low-usage nodes.

policyThe balancing policy. Valid values:
  • datanode: If data is evenly distributed to all DataNodes, your cluster is balanced. This is the default value.
  • blockpool: If all block pools in each DataNode are balanced, your cluster is balanced.
excludeExcludes specified DataNodes.
includeSpecifies DataNodes on which the balancing operation is performed.
sourceThe DataNode that serves as the source node.
blockpoolsThe block pools in which the HDFS Balancer runs.
idleiterationsThe maximum number of idle loops that are allowed. Default value: 5.

Method 2: Run the start-balancer.sh tool

The operation of running the start-balancer.sh tool is equivalent to the operation of running the hdfs daemon start balancer command. To use this tool, perform the following steps:

  1. Log on to a node of the cluster to be configured. For more information, see Log on to a cluster.
  2. Optional:Run the following command to change the maximum bandwidth of the HDFS Balancer:
    hdfs dfsadmin -setBalancerBandwidth <bandwidth in bytes per second>
    Note <bandwidth in bytes per second> specifies the maximum bandwidth. For example, if you want to set the maximum bandwidth to 20 MB/s, set the value of <bandwidth in bytes per second> to 20971520. The complete command is hdfs dfsadmin -setBalancerBandwidth 20971520. If the cluster is heavily loaded, we recommend that you set the value to 209715200, which indicates that the maximum bandwidth is 200 MB/s. If the cluster is idle, we recommend that you set the value to 1073741824, which indicates that the maximum bandwidth is 1 GB/s.
  3. Run the following commands to switch to the hdfs user and run the HDFS Balancer:
    • Data lake cluster
      su hdfs
      /opt/apps/HDFS/hdfs-current/sbin/start-balancer.sh -threshold 5
    • Hadoop cluster
      su hdfs
      /usr/lib/hadoop-current/sbin/start-balancer.sh -threshold 5
  4. Run the following command to go to the installation directory of HDFS:
    • Data lake cluster
      cd /var/log/emr/hadoop/
    • Hadoop cluster
      cd /var/log/hadoop-hdfs
  5. Run the following command to check the status of the HDFS Balancer:
    • Data lake cluster
      tailf hadoop-hdfs-balancer-master-1-1.c-xxx.log
    • Hadoop cluster
      tailf hadoop-hdfs-balancer-emr-header-1.cluster-xxx.log
      Note In the preceding commands, hadoop-hdfs-balancer-master-1-1.c-xxx.log or hadoop-hdfs-balancer-emr-header-xx.cluster-xxx.log is the name of the log file that is obtained in Step 4.

    If the command output includes Successfully, the HDFS Balancer is running.

Parameters used to optimize the performance of the HDFS Balancer

The HDFS Balancer occupies system resources. Therefore, we recommend that you use the HDFS Balancer during off-peak hours. By default, you do not need to modify the parameters of the HDFS Balancer. If you need to modify the parameters of the HDFS Balancer, you can choose Configure > hdfs-site.xml in the HDFS section of the Services tab in the E-MapReduce (EMR) console and adjust the following two types of configurations:
  • Client configurations
    ParameterDescription
    dfs.balancer.dispatcherThreadsThe number of dispatcher threads used by the HDFS Balancer to determine the blocks that need to be moved. Before the HDFS Balancer moves a specific amount of data between two DataNodes, the HDFS Balancer repeatedly retrieves block lists for moving blocks until the required amount of data is scheduled.
    Note Default value: 200.
    dfs.balancer.rpc.per.secThe number of remote procedure calls (RPCs) sent by dispatcher threads per second. Default value: 20.

    Before the HDFS Balancer moves data between two DataNodes, the HDFS Balancer uses dispatcher threads to repeatedly send the getBlocks() RPC to the NameNode. This results in a heavy load on the NameNode. To prevent this issue and balance the cluster load, we recommend that you set this parameter to limit the number of RPCs sent per second.

    For example, you can decrease the value of the parameter by 10 or 5 for a cluster with a high load to minimize the impact on the overall moving progress.

    dfs.balancer.getBlocks.sizeThe total data size of the blocks moved each time. Before the HDFS Balancer moves data between two DataNodes, the HDFS Balancer repeatedly retrieves block lists for moving blocks until the required amount of data is scheduled. By default, the size of blocks in each block list is 2 GB. When the NameNode receives a getBlocks() RPC, the NameNode is locked. If an RPC queries a large number of blocks, the NameNode is locked for a long period of time. This slows down data writing. To prevent this issue, we recommend that you set this parameter based on the NameNode load.
    dfs.balancer.moverThreadsDefault value: 1000.

    The total number of threads that are used to move blocks. Each block move requires a thread.

  • DataNode configurations
    ParameterDescription
    dfs.datanode.balance.bandwidthPerSec

    The bandwidth for each DataNode to balance the workloads of the cluster. We recommend that you set the bandwidth to 100 MB/s. You can also set the dfsadmin -setBalancerBandwidth parameter to adjust the bandwidth. You do not need to restart DataNodes.

    For example, you can increase the bandwidth when the cluster load is low and decrease the bandwidth when the cluster load is high.

    dfs.datanode.balance.max.concurrent.moves

    Default value: 5.

    The maximum number of concurrent block moves that are allowed in a DataNode. Set this parameter based on the number of disks. We recommend that you set this parameter to 4 × Number of disks as the upper limit for a DataNode.

    For example, if a DataNode has 28 disks, set this parameter to 28 on the HDFS Balancer and 112 on the DataNode. Adjust the value based on the cluster load. Increase the value when the cluster load is low and decrease the value when the cluster load is high.

FAQ

Q: Why is the difference approximate to 20% after the balancing even though the threshold parameter is set to 10%?

A: The threshold parameter is used to keep the usage of each DataNode not higher or lower than the average usage of the cluster. Therefore, the difference between the DataNodes with the highest and the lowest usage may be 20% after the balancing. To reduce the difference, you can set the threshold parameter to 5%.