When the EditLog directory on a NameNode or JournalNode grows too large, FsImage checkpoints are failing and EditLog files are accumulating instead of being merged. You may see symptoms such as the EditLog directory growing continuously or NameNode restarts taking much longer than usual.
This article explains why this happens and walks you through restoring normal checkpoint behavior.
How checkpoints work
Hadoop Distributed File System (HDFS) uses two complementary mechanisms to track namespace state:
-
FsImage: a point-in-time snapshot of the entire HDFS namespace
-
EditLog: a running transaction log of every change made since the last snapshot
The NameNode periodically creates a checkpoint by merging the accumulated EditLog transactions into FsImage, then starts a fresh, empty EditLog. This merge keeps EditLog small and makes NameNode restarts fast.
When checkpoints fail, the EditLog can never be merged and keeps growing. The most common causes are:
-
The FsImage directory is full
-
A disk error occurs in the FsImage directory
After a checkpoint failure, HDFS stops writing to the FsImage directory by default and does not retry. This means EditLog accumulates indefinitely until you explicitly re-enable FsImage directory restoration.
A side effect of a large EditLog is significantly longer NameNode restarts, because the NameNode must replay every uncommitted transaction from scratch.
Resolution
Resolve the issue in three steps: immediately re-enable FsImage restoration, optionally reclaim disk space right away, then make the setting permanent.
Step 1: Enable automatic FsImage directory restoration
Run the following commands as the hdfs user:
su - hdfs
hdfs dfsadmin -restoreFailedStorage trueIf the setting takes effect, the output is:
restoreFailedStorage is set to trueThis setting is not persistent. It resets to the default when the NameNode restarts. Complete step 3 to make the change permanent.
After you apply this setting, the NameNode tries to restore the FsImage directory automatically before giving up when the next checkpoint attempt fails.
Step 2 (Optional): Trigger a checkpoint immediately
To reclaim disk space right away without waiting for the next scheduled checkpoint, manually trigger a checkpoint. For instructions, see Manually perform a checkpoint on a NameNode.
The NameNode must enter safe mode during a manual checkpoint, making HDFS unavailable for a short period. Schedule this during a maintenance window.
Completing this step merges the accumulated EditLog files into FsImage immediately, reducing disk usage.
Step 3: Make the setting permanent
To keep the FsImage auto-restore behavior across NameNode restarts, add the configuration property in the EMR console:
-
Go to the E-MapReduce (EMR) console and open your cluster's details page.
-
On the Configure tab of the HDFS service, click hdfs-site.xml.
-
Add the configuration item
dfs.namenode.name.dir.restoreand set the value totrue.
The setting takes effect the next time the NameNode restarts.