This topic describes how to troubleshoot the issue related to the HMaster service of HBase.

Description

The HMaster service of HBase 2.3.4 fails to restart. Error log: 

java.io.EOFException: Cannot seek after EOF
    at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1447)
    at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:65)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)

Cause

When the HMaster service starts, the recovered WAL file is empty. This issue is not likely to occur in most cases.

Issue details of the open source HBase: WAL replay should ignore 0-length files.

Scope of impact

The affected version of HBase is 2.3.4, and clusters of the following E-MapReduce (EMR) versions are affected:
  • Minor versions earlier than EMR V5.6.0
  • EMR V4.8.0 to EMR V4.10.0 (both included)

Solutions

Temporary solution

  1. Find the WAL file whose length is 0 from the exception log and delete the file.
    An exception file usually appears ahead of the file that contains the keyword Failed in the exception log. The exception file contains the keywords regionserver.HRegion: Replaying edits from. log
    For example, you can find the exception WAL file from the preceding error log and run the following command to delete the file: 
    hadoop fs -rm hdfs://emr-header-1.cluster-29****:9000/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/10.154.69.141%2C16000%2C1649314434931.1649326063732
  2. Restart the HMaster service.
    On the Status tab of the HBase service page in the EMR console, find the HMaster service and click Restart in the Actions column.

Permanent solution

Note Back up the JAR file before you replace it.
  1. Download the hbase-server-2.3.4.jar file and replace the JAR file on the master node of the HBase cluster with the downloaded file.
    The JAR file is in the /usr/lib/hbase-current/lib/ directory.
  2. Restart the HMaster service.
    On the Status tab of the HBase service page in the EMR console, find the HMaster service and click Restart in the Actions column.