What do I do if the HMaster service reports the error

HBase 2.3.4 HMaster fails to restart with a java.io.EOFException: Cannot seek after EOF error. This document describes the cause and how to fix it.

Issue

The HMaster service of HBase 2.3.4 fails to restart. The following error appears in the HMaster log:

java.io.EOFException: Cannot seek after EOF
    at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1447)
    at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:65)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)

Affected versions

This issue affects HBase 2.3.4 on the following E-MapReduce (EMR) versions:

EMR minor versions earlier than V5.6.0
EMR V4.8.0 to V4.10.0 (inclusive)

Cause

When HMaster starts, it reads recovered Write-Ahead Log (WAL) files from Hadoop Distributed File System (HDFS). If a recovered WAL file has zero bytes, HBase 2.3.4 throws an EOFException instead of skipping the file.

This edge case is tracked in the upstream open-source issue HBASE-25053: WAL replay should ignore 0-length files.

Note

This issue is rare in normal operations. It occurs only when a recovered WAL file is left with zero bytes.

Solution

Replace the hbase-server-2.3.4.jar file on the master node with a patched version, then restart the HMaster service.

Important

Back up the existing JAR file before replacing it.

Download the patched JAR file: hbase-server-2.3.4.jar
On the master node, replace the existing JAR file with the downloaded file. The JAR file is located at /usr/lib/hbase-current/lib/.
Restart the HMaster service. In the EMR console, go to the HBase service page. On the Status tab, find the HMaster service and click Restart in the Actions column.