What do I do if a

When Mismatched address stored in ZK appears in the ZooKeeper Failover Controller (ZKFC) logs of a high availability (HA) cluster, Hadoop Distributed File System (HDFS) cannot elect an active NameNode.

Error message

java.lang.RuntimeException: Mismatched address stored in ZK for NameNode at /**.**.**.**:8020: Stored protobuf was nameserviceId: "hdfs-cluster"
namenodeId: "nn1"
port:8020
zkfcPort:8019
,address from our own configuration for this NameNode was xxxxx.emr.aliyuncs.com/**.**.**.**:8021

Cause

If the NameNode information stored in the ZKFC component of an HA cluster does not match the Hadoop configuration file, HDFS cannot elect an active NameNode. ZK holds the RPC port as 8020, but the Hadoop configuration sets the NameNode ServiceRPC port to 8021. In earlier versions of E-MapReduce (EMR), the ServiceRPC feature is disabled by default. Manually setting the ServiceRPC port to 8021 creates this mismatch and requires you to re-initialize ZKFC.

Solution 1: Re-initialize ZKFC

Stop ZKFC, reinitialize the ZK data, and restart ZKFC.

Stop the ZKFC component in the EMR console. On the Services tab of your cluster, click Status in the HDFS section. On the Status tab, find the ZKFC component and click Stop in the Actions column.
Log on to the cluster and run the following commands to reinitialize ZKFC:
```
su - hdfs
hdfs zkfc -formatZK
```
Start the ZKFC component in the EMR console. On the Services tab of your cluster, click Status in the HDFS section. On the Status tab, find the ZKFC component and click Start in the Actions column.

Solution 2: Remove the ServiceRPC configuration

Clear the ServiceRPC parameters and restart HDFS to restore the address stored in ZK to match the default RPC port.

Clear the ServiceRPC configuration in the EMR console. On the Services tab of your cluster, click Configure in the HDFS section. On the hdfs-site.xml tab, leave the following parameters blank:

dfs.namenode.servicerpc-address.hdfs-cluster.nn1
dfs.namenode.servicerpc-address.hdfs-cluster.nn2
dfs.namenode.servicerpc-address.hdfs-cluster.nn3  (Hadoop 3 only)

Or, if your cluster uses the emr-cluster nameservice ID:

dfs.namenode.servicerpc-address.emr-cluster.nn1
dfs.namenode.servicerpc-address.emr-cluster.nn2
dfs.namenode.servicerpc-address.emr-cluster.nn3  (Hadoop 3 only)

Restart HDFS in the EMR console. On the Services tab of your cluster, click Status in the HDFS section. Choose More > Restart in the upper-right corner.