HMaster service fails with

Resolve HBase HMaster startup failures caused by the "Cannot seek after EOF" error in HBase 2.3.4.

Symptoms

HMaster for HBase 2.3.4 fails to restart with the following error in the logs:

java.io.EOFException: Cannot seek after EOF
    at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1447)
    at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:65)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)

Cause

The HMaster service fails to start because it encounters an empty (0-byte) write-ahead log (WAL) file during its recovery process. This issue is rare.

Related community issue: WAL replay should ignore 0-length files.

Affected versions

This issue affects HBase version 2.3.4 on EMR clusters running the following EMR versions:

EMR versions earlier than EMR-5.6.0
EMR-4.8.0 to EMR-4.10.0 (inclusive)

Resolution

Note

Back up the JAR file before you replace it.

Download hbase-server-2.3.4.jar and replace the corresponding JAR file on the master node of the HBase cluster.

The JAR file is in the /usr/lib/hbase-current/lib/ directory.
Restart the HMaster service.

In the EMR console, go to the Status page of the HBase service. In the HMaster component row, click Restart in the Actions column.