Diagnose ZooKeeper Instability and Data Migration Issues - E-MapReduce - Alibaba Cloud - E-MapReduce

Common questions about ZooKeeper service stability, connection limits, and data directory migration.

What to do if the ZooKeeper service is unstable or restarts unexpectedly?
How to smoothly migrate the ZooKeeper data directory?

Service instability or unexpected restarts

The most common cause of instability is a high znode count or large snapshot size. Because ZooKeeper keeps all znodes in memory and synchronizes data across nodes, excessive znodes or snapshots degrade stability. As a best practice, keep the znode count below 100,000 and the snapshot size below 800 MB.

You can view the znode count for your ZooKeeper cluster on the Monitoring tab in the E-MapReduce console.
Check the snapshot size.
1. On the Configure tab of the ZooKeeper service, find the value of the dataDir parameter to get the ZooKeeper data directory.
2. Run the following command to check the size of snapshots in the ZooKeeper data directory.
```
ls -lrt /mnt/disk1/zookeeper/data/version-2/snapshot*
```
  If the number of znodes or the snapshot size is too large, investigate the znode distribution to prevent related applications from overusing ZooKeeper.

Migrate the data directory

If you need to change the ZooKeeper data directory due to insufficient disk space or poor disk performance, migrate the directory on each node individually. This approach avoids stopping the entire ZooKeeper service.

Note For example, this procedure changes the data directory from /mnt/disk1/zookeeper/data to /mnt/disk2/zookeeper/data. In this example cluster, the master-1-2 node is the leader, and the master-1-1 and master-1-3 nodes are followers. Migrate the followers first, and then the leader.

Modify the data directory and save the configuration.
1. On the Configure tab of the ZooKeeper service, search for the dataDir parameter and change its value to the new directory, /mnt/disk2/zookeeper/data.
2. Click Save.
3. In the dialog box that appears, enter an Execution Reason and click Save.
Deploy the configuration.
1. On the Configure tab of the ZooKeeper service, click Deploy Client Configuration.
2. In the dialog box that appears, enter the Execution Reason and click OK.
3. In the Confirm dialog box, click OK.
Optional: Verify the data directory.
1. Connect to the cluster over SSH. For more information, see Connect to a cluster.
2. Run the following command to view the value of the dataDir parameter in the zoo.cfg configuration file.
```
 cat /etc/emr/zookeeper-conf/zoo.cfg
```
  The output shows that the data directory has been updated.
Stop the master-1-1 node.
1. On the Status tab, click STOP in the Actions column for the master-1-1 node.
2. In the Execute Cluster Action dialog box, enter the Execution Reason and click OK.
3. In the Confirm dialog box, click OK.
Migrate the directory.
1. Connect to the cluster over SSH. For more information, see Connect to a cluster.
2. Run the following command on the master-1-1 node to migrate the directory and set the required permissions.
```
sudo rm -rf /mnt/disk2/zookeeper && sudo cp -rf /mnt/disk1/zookeeper /mnt/disk2/zookeeper && sudo chown hadoop:hadoop -R /mnt/disk2/zookeeper
```
Start the master-1-1 node.
1. On the Status tab, click START in the Actions column for ZookeeperServer.
2. In the dialog box that appears, enter the Execution Reason and click OK.
3. In the Confirm dialog box, click OK.
  Refresh the page until the Health Status of the master-1-1 node changes to Good.
Log on to the master-1-3 node and repeat Step 4 to Step 6.
Log on to the master-1-2 node and repeat Step 4 to Step 6.
The migration is complete when all nodes are healthy.
Note The master-1-2 node was originally the leader. After you stop it, it becomes a follower, and one of the other two nodes is elected as the new leader.