E-MapReduce:HDFS optimization

Control the number of small files

The HDFS NameNode keeps metadata for every file in memory. When a large number of small files accumulate, the NameNode hits a memory bottleneck even if total disk capacity is not exhausted.

Merge existing small files into fewer, larger files to reduce NameNode memory pressure.

Limit files per directory

Spark, YARN, and other services can continuously write files to the same HDFS directory. HDFS enforces a maximum number of files per directory, and exceeding the limit causes job failures.

Plan your data layout in advance — classify files by time period or business type so no single directory grows unbounded. Set dfs.namenode.fs-limits.max-directory-items to define the limit. A value of 1,000,000 works well for most workloads.

To add the parameter:

1. In the EMR console, go to the HDFS service page and click the Configure tab.

2. Click the hdfs-site tab.

3. Click Add Configuration Item, add dfs.namenode.fs-limits.max-directory-items, and set the value.

4. Click Save.

For detailed steps, see Manage configuration items.

Tolerate failed DataNode volumes

By default, if multiple data storage volumes are configured for a DataNode and one of the volumes fails, the DataNode stops providing services. By default, at least one available volume is configured.

Raising the threshold lets a DataNode continue operating as long as the number of failed volumes stays within the configured limit (dfs.datanode.failed.volumes.tolerated = 0 by default).

To change the threshold:

1. In the EMR console, go to the HDFS service page and click the Configure tab.

2. Search for dfs.datanode.failed.volumes.tolerated and update the value.

Balance DataNode capacity

Adding DataNodes to an EMR cluster can leave data unevenly distributed across nodes, causing some DataNodes to become overloaded. Run the HDFS balancer to redistribute data.