How to troubleshoot the issue that a NameNode cannot exit the safe mode after the NameNode is started - E-MapReduce

This topic describes how to troubleshoot the issue that a NameNode cannot exit the safe mode after the NameNode is started.

Error message

The following error message in the NameNode logs or the web user interfaces (UIs) of Hadoop Distributed File System (HDFS) components indicates that the NameNode cannot exit the safe mode. As a result, you cannot use most services of HDFS:

Safemode is ON.The reported blocks xxx needs addition ablocks to reach the threshold 0.9990 of total blocks yyy

Cause

If the number of blocks reported by DataNodes does not reach the specified threshold, the NameNode cannot exit the safe mode. The threshold is the proportion of the reported blocks in the total blocks. The default threshold is 0.999f. Check whether all DataNodes are started. If all the DataNodes are started, check whether the following scenarios occur:

Some blocks are lost due to improper O&M and the threshold cannot be reached. In this scenario, the NameNode cannot automatically exit the safe mode.
When a large number of HDFS processes, such as the DataNodes or NameNodes, are restarted due to exceptions rather than normal HA switchovers, it takes a long time for the DataNodes to report the blocks. In this scenario, the DataNodes need to trigger the report of full blocks, and the NameNode can automatically exit the safe mode only when the number of received blocks reaches the threshold. The duration of the entire process depends on the number of blocks.

Solutions

Solution 1: Run the following command as the hdfs user to manually exit the safe mode:
```
hdfs dfsadmin -safemode leave
```
Solution 2: Run the following command to decrease the threshold. After you decrease the threshold, you need to perform the following steps to make the modification take effect for the NameNode: On the Status tab in the HDFS section, click Restart in the Actions column of the NameNode component.
```
dfs.namenode.safemode.threshold-pct=0.9f (Default value: 0.999f.)
```
Note
In the preceding example, 0.9f is used. You can specify the value of the dfs.namenode.safemode.threshold-pct parameter based on your business requirements.