How to troubleshoot the issue that a "Mismatched address stored in ZK" error occurs in the ZKFC logs of an HA cluster - E-MapReduce

This topic describes how to troubleshoot the issue that a Mismatched address stored in ZK error occurs in the ZooKeeper Failover Controller (ZKFC) logs of a high availability (HA) cluster. If this error occurs, Hadoop Distributed File System (HDFS) cannot elect an active NameNode.

Error message

java.lang.RuntimeException: Mismatched address stored in ZK for NameNode at /**.**.**.**:8020: Stored protobuf was nameserviceId: "hdfs-cluster"
namenodeId: "nn1"
port:8020
zkfcPort:8019
,address from our own configuration for this NameNode was xxxxx.emr.aliyuncs.com/**.**.**.**:8021

Cause

If the NameNode information stored in the ZKFC component of an HA cluster does not match the configuration file of Hadoop, HDFS cannot elect an active NameNode. The remote procedure call (RPC) port stored in the ZKFC component is 8020, but the NameNode ServiceRPC port is 8021. In E-MapReduce (EMR) of some earlier versions, the ServiceRPC feature is disabled. If you manually set the ServiceRPC port to 8021, you must initialize the ZKFC component.

Solutions

Solution 1

Stop the ZKFC component by using the EMR console.
On the Services tab of the cluster that you want to manage, click Status in the HDFS section. Then, find the ZKFC component on the Status tab and click Stop in the Actions column.
Log on to the cluster. Run the following commands to initialize the ZKFC component:
```
su - hdfs
hdfs zkfc -formatZK
```
Start the ZKFC component by using the EMR console.
On the Services tab of the cluster that you want to manage, click Status in the HDFS section. Then, find the ZKFC component on the Status tab and click Start in the Actions column.

Solution 2

Delete the original ServiceRPC configurations by using the EMR console.

On the Services tab of the cluster that you want to manage, click Configure in the HDFS section. On the hdfs-site.xml tab, leave the following parameters empty:

dfs.namenode.servicerpc-address.hdfs-cluster.nn1
dfs.namenode.servicerpc-address.hdfs-cluster.nn2
dfs.namenode.servicerpc-address.hdfs-cluster.nn3 (For Hadoop 3 only)

Or

dfs.namenode.servicerpc-address.emr-cluster.nn1
dfs.namenode.servicerpc-address.emr-cluster.nn2
dfs.namenode.servicerpc-address.emr-cluster.nn3 (For Hadoop 3 only)

Restart HDFS by using the EMR console.
On the Services tab of the cluster that you want to manage, click Status in the HDFS section. Choose More > Restart in the upper-right corner to restart HDFS.