Perform disk O&M for EMR Kafka 2.4.1 - E-MapReduce - Alibaba Cloud Documentation Center

In most cases, disk failures and O&M involve data destruction. Before you perform disk O&M, you must consider whether to migrate or back up data. For Kafka clusters, you must also consider whether partition replicas of a Kafka topic can be synchronized from partition replicas on other brokers. This topic describes how to perform O&M after disk failures occurred in E-MapReduce (EMR) Kafka clusters. In this topic, Kafka 2.4.1 is used.

Business scenarios

Kafka stores log data in disks. Disk failures can lead to various issues, such as decreased I/O performance, reduced cluster throughput, increased message read and write latency, or log directories going offline. These issues may affect the continuity of online business, cause data loss, and weaken the fault tolerance capabilities of a Kafka cluster. The failure of a single disk may cause an avalanche effect in the Kafka cluster and lead to severe production issues due to decreased I/O capabilities. To discover failures at the earliest opportunity, you can configure metrics to monitor disks. After a disk fails, you must promptly handle the failure to recover the fault tolerance capabilities of the Kafka cluster.

Overview

This topic describes the disk O&M policies from the aspects of disk monitoring and disk failure recovery.

Disk monitoring

This section describes the disk monitoring policies from the perspectives of Kafka clusters and Elastic Compute Service (ECS) instances.

Kafka cluster: You can configure metrics, such as OfflineLogDirectoryCount and UnderReplicatedPartitions, for an EMR Kafka cluster in CloudMonitor to detect exceptions at the earliest opportunity.
ECS instance: You can configure metrics, such as metrics related to I/O wait time and bandwidth, for ECS instances in CloudMonitor to monitor the health status of disks. ECS automatically checks the disk status and immediately pushes related disk events to you after exceptions are detected.

Disk failure recovery

If the OfflineLogDirectoryCount or UnderReplicatedPartitions metric is abnormal, check whether the exception is caused by a disk failure at the earliest opportunity.

After a disk failure occurs, you must select a recovery solution based on various factors, such as the cause and impact of the failure, tolerance to data loss and long-time business unavailability, and cluster status.

If service availability is the top priority and data loss is tolerable, you can consider restoring the Kafka service first, even if it means potentially losing data.
If data loss is unacceptable but long-time service unavailability is tolerable, you must select a solution that prevents data loss.
If high availability and data integrity are both required, you can modify the configuration of the Kafka cluster to improve the fault tolerance. This helps prevent data loss caused by a single disk failure.
If the overall performance of a Kafka cluster is decreased due to a decrease in I/O performance of a faulty disk, and your business is affected, you must immediately isolate the faulty disk.

After a disk failure is detected, you can use one of the following solutions to perform disk O&M:

Partition migration between brokers: You can migrate the partition replicas on the faulty disk to a disk of another broker. For more information, see Partition migration between brokers.
Partition migration within a broker: You can migrate the partition replicas on the faulty disk to another disk that is attached to the same broker. For more information, see Partition migration within a broker.
Data restoration on the broker: You can delete the faulty disk from the log.dirs configuration item and restart the broker. The lost partition data is automatically restored to another directory on the broker. For more information, see Data restoration on a broker.

Limits

This topic applies only to clusters of EMR V5.8.0 or a later minor version and clusters of EMR V3.42.0 or a later minor version.

Precautions

When you perform disk O&M, take note of the following items:

Select a solution based on your business requirements.
If data replication and migration are involved during disk O&M, you must specify a throttling threshold for O&M traffic to protect normal business traffic from being affected.
Specify an appropriate throttling threshold for migration traffic based on the I/O capabilities of the disk and normal business traffic.
After the physical disk is repaired and re-attached to the broker, you must migrate the partition data of the topic back to the disk and migrate the leader load back to the broker.
If data replication and migration are involved during disk O&M, you must evaluate the impact of the data migration duration on your business.

Partition migration between brokers

Solution description

If a disk fails or needs to be replaced due to specific reasons, you can migrate the partition replicas on the faulty disk to another disk. During disk repair or replacement, the number of partition replicas remains unchanged and the fault tolerance of the cluster is unaffected. After disk repair or replacement, you must migrate the partition data back to the repaired disk or to the new disk. The advantage of the solution lies in unaffected fault tolerance of the cluster and balanced load among brokers during disk O&M. The disadvantage is that data loss may occur when the partitions on the faulty disk have no in-sync replicas (ISRs).

Scenarios

The broker to which the faulty disk is attached does not have sufficient disk space or is overloaded.

Precautions

Migrating partition data requires a large amount of traffic. You must specify a throttling threshold for migration traffic to protect normal business from being affected.
The time required for data migration varies based on the amount of data that you want to migrate.
You must specify an appropriate throttling threshold for migration traffic based on the I/O capabilities of the disk and normal business traffic.
Make sure that the cluster has sufficient disk space to store the migrated data.
The original assignment file must be backed up. Before you migrate partitions, you must save the original assignment file on the faulty disk. This helps you migrate the partitions back to the original directory on the repaired disk.

disk1 stores the log files of Kafka. You must perform the following operations before and after disk O&M.

Before you detach disk1 from a broker, migrate the log files to another directory.

# Create a new log directory. Make sure that the permissions on the new directory are the same as the permissions on the /mnt/disk1/log/ directory. 
sudo mkdir /mnt/disk2/log

# Check whether the symbolic link points to disk2. 
sudo ln -sf /mnt/disk2/log /var/log/taihao-apps

# Migrate the log files stored in disk1 to the new log directory. 
sudo mv /mnt/disk1/log/* /mnt/disk2/log/

After you reattach disk1 to the broker, migrate the log files back to disk1.

# Create a log directory for disk1. 
sudo mkdir /mnt/disk1/log

# Check whether the symbolic link points to disk1. 
sudo ln -sf /mnt/disk1/log /var/log/taihao-apps

# Move the log files back to disk1. 
sudo mv /mnt/disk2/log/* /mnt/disk1/log/

Procedure

This section describes how to migrate partitions on a faulty disk from a broker to another broker to implement disk O&M. In this example, the faulty disk is attached to Broker 0 and the directory of the disk is /mnt/disk7.

Migrate partitions.
1. Log on to the master node of the Kafka cluster in SSH mode. For more information, see Log on to a cluster.
2. Optional. Run the following command to isolate the faulty disk:
```
chmod 000 /mnt/disk7
```
3. Run the following command to migrate the partitions on the faulty disk to another broker:
```
kafka-reassign-partitions.sh --zookeeper  master-1-1:2181/emr-kafka --reassignment-json-file reassign.json --throttle 30000000 --execute
```
  Note
  The reassign.json file contains information about the partition replicas that need to be migrated.
  After the command is run, the original assignment file is generated. You must save the file for subsequent data restoration.
4. Run the following command to check whether data migration from the faulty disk is complete:
```
ls -lrt /mnt/disk7/kafka/log
```
  Check whether the log directory contains log files based on the command output. If no log files exist, data migration is complete.
5. Optional. If the faulty disk is not isolated, run the following command to isolate the disk:
```
chmod 000 /mnt/disk7
```
Unmount the directory of the faulty disk.
Important
When you edit the fstab file, you must delete the directory of the faulty disk rather than commenting out the directory.
1. Run the following command to edit the /etc/fstab file:
```
vim /etc/fstab
```
2. Delete the disk7 directory.
3. Run the following command to check whether the directory handle is released:
```
lsof +D /mnt/disk7
```
  If the handle is not released after more than 30 minutes, restart the broker in the console.
4. Run the following command to unmount the directory of the faulty disk:
```
umount /mnt/disk7
```
Modify the log.dirs configuration item and restart the broker.
1. In the EMR console, modify the log.dirs configuration item for the broker and delete the partition directory of the faulty disk.
  In this example, the /mnt/disk7/kafka/log directory is deleted.
2. Restart the broker in the EMR console. For more information, see Restart a service.
  Note
  If the O&M process for disk repair events requires restarting ECS instances, you can restart the broker and the instances at the same time.
Complete the disk repair in the ECS console.
The repair may require several days.
Attach the repaired disk. For more information, see the Mount the restored disk to the ECS instance section of the "Restore disks of ECS instances for brokers in EMR Kafka clusters" topic.
You must attach the repaired disk to the original directory. In this example, the disk is attached to the /mnt/disk7 directory.

Create a Kafka log directory in the directory of the repaired disk.

// Create a log directory. In this example, a log directory named /mnt/disk7/kafka/log is created. 
sudo mkdir -p /mnt/disk7/kafka/log
sudo chown -R kafka:hadoop /mnt/disk7/kafka/log

Modify the log.dirs configuration item and restart the broker.
1. In the EMR console, modify the log.dirs configuration item for the broker whose faulty disk is repaired and add the partition directory of the repaired disk.
  In this example, the /mnt/disk7/kafka/log directory is added.
2. Restart the broker in the EMR console. For more information, see Restart a service.
Migrate the partition replicas back to the repaired disk.
Use the original assignment file saved in Step 1 to migrate the partition replicas on the faulty disk to the repaired disk.

Data restoration on the broker

Solution description

If the I/O performance of a faulty disk significantly decreases, you must isolate the disk to prevent a single point of failure (SPOF) from affecting cluster performance.

After the faulty disk is isolated, the status of the Kafka log directory of the disk changes to offline. If the partitions on the faulty disk have ISRs or partition data loss is tolerable, you can delete the directory of the faulty disk from the log.dirs configuration item and then restart the broker. If a new leader replica exists on another broker after you restart the broker, the original partition replicas of the faulty disk are restored to the broker on which the leader replica exists. After you attach the repaired disk to the broker, you can use the reassign tool to migrate the original data back to the repaired disk.

The disadvantage of the solution is that data loss may occur when the partitions on the faulty disk have no ISRs.

Scenarios

The I/O performance of a faulty disk significantly decreases and the faulty disk must be isolated at the earliest opportunity.
The log directory of the faulty disk is offline.
If the partitions on the faulty disk have no ISRs on other brokers, partition data loss is tolerable.

Precautions

When data is restored on the same broker, replica restoration traffic is generated. You must specify a throttling threshold based on your business requirements.
Make sure that other brokers have sufficient space to store data on the faulty disk.
After you reattach the repaired disk, the data must be migrated from other disks back to the repaired disk to ensure balanced load among the disks.

Procedure

This section describes how to restore data on a faulty disk to implement disk O&M. In this example, the faulty disk is attached to Broker 0 and the directory of the disk is /mnt/disk7.

Isolate the faulty disk.
1. Log on to the master node of the Kafka cluster in SSH mode. For more information, see Log on to a cluster.
2. Run the following command to isolate the faulty disk:
```
chmod 000 /mnt/disk7
```
Optional. Back up the name of the Kafka log directory on the faulty disk.
After the disk is repaired and reattached, you can migrate the original partitions back to the repaired disk based on the name of the Kafka log directory.
Unmount the directory of the faulty disk.
Important
When you edit the fstab file, you must delete the directory of the faulty disk rather than commenting out the directory.
1. Run the following command to edit the /etc/fstab file:
```
vim /etc/fstab
```
2. Delete the disk7 directory.
3. Run the following command to check whether the directory handle is released:
```
lsof +D /mnt/disk7
```
  If the handle is not released after more than 30 minutes, restart the broker in the console.
4. Run the following command to unmount the directory of the faulty disk:
```
umount /mnt/disk7
```
Modify the log.dirs configuration item of the broker with the faulty disk and specify a throttling threshold for data restoration traffic.
1. In the EMR console, modify the log.dirs configuration item for the broker to delete the partition directory of the faulty disk.
  In this example, the /mnt/disk7/kafka/log directory is deleted.
2. Optional. Configure a specific throttling parameter to limit the traffic bandwidth that is used to reassign partitions to implement data restoration on the broker. For more information, see Throttle the O&M traffic of Kafka brokers.
Restart the broker in the EMR console. For more information, see Restart a service.
Note
If the O&M process for disk repair events requires restarting ECS instances, you can restart the broker and the instances at the same time.
During the restart, Kafka restores the partition replicas missing on the faulty disk to other disks. If you configured a throttling parameter, delete the parameter after Kafka restores the partition data.
In the ECS console, complete the disk repair operations.
The repair may require several days.
Attach the repaired disk. For more information, see the Mount the restored disk to the ECS instance section of the "Restore disks of ECS instances for brokers in EMR Kafka clusters" topic.
You must attach the repaired disk to the original directory. In this example, the disk is attached to the /mnt/disk7 directory.
Create a Kafka log directory and restart the broker.
1. Create a Kafka log directory in the directory of the repaired disk.
```
// Create a log directory. In this example, a log directory named /mnt/disk7/kafka/log is created. 
sudo mkdir -p /mnt/disk7/kafka/log
sudo chown -R kafka:hadoop /mnt/disk7/kafka/log
```
2. In the EMR console, modify the log.dirs configuration item for the broker whose faulty disk is repaired and add the partition directory of the repaired disk.
  In this example, the /mnt/disk7/kafka/log directory is added.
3. Restart the broker in the EMR console. For more information, see Restart a service.
Migrate the partition replicas back to the repaired disk.
Use the partition replica directory saved in Step 2 and the reassign tool to migrate the internal directory of the broker on which partition replicas reside to the repaired disk.

Partition migration within a broker

Solution description

If the I/O performance of a faulty disk significantly decreases, you must isolate the disk to prevent the SPOF from affecting cluster performance.

After the faulty disk is isolated, the status of the Kafka log directory of the disk changes to offline. If a replica on the faulty disk cannot be migrated to another broker and data loss is not tolerable, you can migrate the partition replicas on the faulty disk to another disk within the current broker. Then, you can perform subsequent disk O&M operations. This solution is suitable for scenarios in which data loss is not tolerable.

Scenarios

The partitions on the faulty disk have no ISRs on other brokers and partition data loss is not tolerable.

Precautions

You must evaluate the impact of I/O hotspots that occur when the mv commands are run in the operating system.

Procedure

This section describes how to migrate partitions on a faulty disk to another disk on the same broker to implement disk O&M. In this example, the faulty disk is attached to Broker 0 and the directory of the disk is /mnt/disk7.

Isolate the faulty disk.
1. Log on to the master node of the Kafka cluster in SSH mode. For more information, see Log on to a cluster.
2. Run the following command to isolate the faulty disk:
```
chmod 000 /mnt/disk7
```
Optional. Back up the name of the Kafka log directory on the faulty disk.
After the disk is repaired and reattached, you can migrate the original partitions back to the repaired disk based on the name of the Kafka log directory.
Migrate the partitions within the broker.
You can migrate the partitions in the /mnt/disk7/kafka/log directory to other directories on the current broker based on the disk space occupied by each partition. For more information, see the Migrate partitions within a broker section in the "Perform O&M operations when the disk space of an EMR Kafka cluster is full" topic.
Unmount the directory of the faulty disk.
Important
When you edit the fstab file, you must delete the directory of the faulty disk rather than commenting out the directory.
1. Run the following command to edit the /etc/fstab file:
```
vim /etc/fstab
```
2. Delete the disk7 directory.
3. Run the following command to check whether the directory handle is released:
```
lsof +D /mnt/disk7
```
  If the handle is not released after more than 30 minutes, restart the broker in the console.
4. Run the following command to unmount the directory of the faulty disk:
```
umount /mnt/disk7
```
Restart the broker in the EMR console. For more information, see Restart a service.
Note
If the O&M process for disk repair events requires restarting ECS instances, you can restart the broker and the instances at the same time.
Complete the disk repair in the ECS console.
The repair may require several days.
Attach the repaired disk. For more information, see the Mount the restored disk to the ECS instance section of the "Restore disks of ECS instances for brokers in EMR Kafka clusters" topic.
You must attach the repaired disk to the original directory. In this example, the disk is attached to the /mnt/disk7 directory.
Create a Kafka log directory and restart the broker.
1. Create a Kafka log directory in the directory of the repaired disk.
```
// Create a log directory. In this example, a log directory named /mnt/disk7/kafka/log is created. 
sudo mkdir -p /mnt/disk7/kafka/log
sudo chown -R kafka:hadoop /mnt/disk7/kafka/log
```
2. In the EMR console, modify the log.dirs configuration item for the broker whose faulty disk is repaired and add the partition directory of the repaired disk.
3. Restart the broker in the EMR console. For more information, see Restart a service.
Optional. Migrate the partition replicas back to the repaired disk.
Use the partition replica directory saved in Step 2 and the reassign tool to migrate the internal directory of the broker on which partition replicas reside to the repaired disk.