What do I do if the HMaster service reports the error "Cannot seek after EOF" - E-MapReduce

This topic describes how to troubleshoot the issue related to the HMaster service of HBase.

Description

The HMaster service of HBase 2.3.4 fails to restart. Error log:

java.io.EOFException: Cannot seek after EOF
    at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1447)
    at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:65)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)

Cause

When the HMaster service starts, the recovered WAL file is empty. This issue is not likely to occur in most cases.

Issue details of the open source HBase: WAL replay should ignore 0-length files.

Scope of impact

The affected version of HBase is 2.3.4, and clusters of the following E-MapReduce (EMR) versions are affected:

Minor versions earlier than EMR V5.6.0
EMR V4.8.0 to EMR V4.10.0 (both included)

Solution

Note

Back up the JAR file before you replace it.

Download the hbase-server-2.3.4.jar file and replace the JAR file on the master node of the HBase cluster with the downloaded file.
The JAR file is in the /usr/lib/hbase-current/lib/ directory.
Restart the HMaster service.
On the Status tab of the HBase service page in the EMR console, find the HMaster service and click Restart in the Actions column.