Tune the JVM memory for NameNode and DataNode to improve HDFS stability.
Adjust the NameNode JVM memory
-
Background: In HDFS, the NameNode stores file metadata, which consumes memory. The default JVM configuration handles common HDFS workloads, but workloads with a large or growing number of files may exceed the default memory capacity. In this case, increase the allocated JVM memory.
-
Recommendation: Adjust the NameNode JVM memory size as follows:
-
For an HA cluster
In the EMR console, navigate to the Configure page of the HDFS service. Search for the hadoop_namenode_heapsize parameter and set its value as needed.
-
For a non-HA cluster
In the EMR console, navigate to the Configure page of the HDFS service. Search for the hadoop_namenode_heapsize and hadoop_secondary_namenode_heapsize parameters and set their values as needed.
NoteTo apply the changes, restart the corresponding NameNode or SecondaryNameNode service.
You can view the number of files (Files) and blocks (Blocks) on the HDFS web UI. For information about accessing the web UI, see Access the web UIs of open source components. Use the following formula to calculate the NameNode JVM memory size.
Recommended value (MB) = (Number of files in millions + Number of blocks in millions) × 512For example, if you have 10 million small- to medium-sized files where each file does not exceed one block, the block count is also 10 million. The recommended memory size is (10 + 10) × 512 = 10,240 MB.
The following table lists recommended values for scenarios where most files are no larger than a single block.
Number of files
Recommended value (MB)
10,000,000
10240
20,000,000
20480
50,000,000
51200
100,000,000
102400
-
Adjust the memory size of the JVM for each DataNode
- Background information: In HDFS, DataNodes record metadata of data blocks. The metadata occupies a specific amount of memory space. The default JVM memory configuration can meet some common requirements of HDFS. Sometimes more files are written to HDFS and the number of files stored in HDFS continuously increases. The number of blocks on DataNodes depends on the number of files. When the number of newly written files exceeds the default memory space configuration, these files cannot be stored. In this case, you must modify the memory size.
- Suggestion: Go to the Configure tab on the HDFS service page in the EMR console. Search for the hadoop_datanode_heapsize parameter in the Configuration Filter section and change the value of this parameter based on your business requirements. You can use the following formula to calculate the memory size of the JVM for each DataNode:
Number of replicas per DataNode in an EMR cluster = Number of data blocks × 3/Number of DataNodes Recommended memory size = Number of replicas per DataNode, measured in millions × 2048 MBNote After the configuration is complete, you must restart DataNodes to apply the configuration.For example, HDFS in an EMR cluster provides triplicate storage, the Elastic Compute Service (ECS) instances deployed in the EMR cluster belong to a big data instance family, and the number of core nodes is 6. If you have 10 million files and the number of data blocks is also 10 million because all the files are small- and medium-sized files, the number of replicas per DataNode is 5,000,000, which is calculated by using the following formula: (10,000,000 × 3/6). The recommended memory size is 10,240 MB, which is calculated by using the following formula: 5 × 2048 MB.
The following table describes the mapping between the number of replicas per DataNode and the recommended memory size. The size of most files does not exceed the size of a block.Number of replicas per DataNode Recommended memory size (MB) 1,000,000 2048 2,000,000 4096 5,000,000 10240 Note The recommended value includes the memory space for the JVM kernel and the memory space to run jobs during peak hours. You can directly use the recommended value under normal circumstances.