What do I do if a NameNode cannot exit the safe mode after the NameNode is started?

Updated at:
Copy as MD

When a NameNode stays in safe mode after startup, most Hadoop Distributed File System (HDFS) operations are blocked. This topic explains why this happens and how to resolve it.

Error message

The following message appears in the NameNode logs or the HDFS web UI:

Safemode is ON.The reported blocks xxx needs addition ablocks to reach the threshold 0.9990 of total blocks yyy

How safe mode works

On startup, the NameNode enters safe mode and waits for DataNodes to report their blocks. Once the proportion of reported blocks reaches the configured threshold (default: 0.999f), the NameNode exits safe mode automatically.

If the reported block count never reaches the threshold, the NameNode stays in safe mode indefinitely.

Diagnose the cause

Confirm that all DataNodes are running. If all DataNodes are up but the NameNode is still in safe mode, identify which scenario applies:

Scenario Description Can auto-exit?
Block loss Blocks were lost due to improper O&M. The threshold cannot be reached even after all DataNodes report in. No — manual action required
Slow block reporting A large number of HDFS processes (DataNodes or NameNodes) restarted due to exceptions rather than normal HA switchovers. DataNodes are still reporting blocks. Yes — waits until reported block count reaches the threshold

Solution 1: Manually exit safe mode

Use this solution when block loss has occurred and the NameNode cannot exit safe mode automatically.

Run the following command as the hdfs user:

hdfs dfsadmin -safemode leave

Solution 2: Lower the safe mode threshold

If your cluster has permanently lost a small number of blocks but the remaining data is intact, lower the threshold so that the NameNode can exit safe mode.

  1. Set the following parameter in the HDFS configuration:

    dfs.namenode.safemode.threshold-pct=0.9f

    The default value is 0.999f. The example uses 0.9f; set the value based on your business requirements.

  2. On the Status tab in the HDFS section, click Restart in the Actions column of the NameNode component to apply the change.