Test service performance when the disk I/O load is high in ECS by performing a high I/O load drill - Elastic Compute Service

An Elastic Compute Service (ECS) instance may experience the high I/O load issue due to factors such as excessive business load, unexpected non-business process occupation, or insufficient memory resources. The issue may cause business performance degradation and risks such as data loss. This topic describes how to perform a drill to verify that a disk high I/O load issue can be injected and resolved.

Implementation

The Cloud Assistant plug-in named ACS-ECS-HighIo uses the built-in data duplicator (dd) tool to copy data to and convert a file to increase the disk I/O load.

When the plug-in works in read mode, the dd tool writes a temporary file to a destination path and repeatedly reads the file to simulate the sustained read pressure.
When the plug-in works in write mode, the dd tool continuously writes data to the file to simulate the sustained write pressure.

Procedure

Prerequisites

Cloud Assistant Agent is installed on the ECS instance for which you want to perform a drill.
The status of Cloud Assistant is Normal on the ECS instance. For more information, see View the status of Cloud Assistant and handle anomalies.

Fault injection

Log on to an ECS instance.
For more information, see Use Workbench to connect to a Linux instance over SSH.
Use the sudo user to run the ACS-ECS-HighIo plug-in.
```
sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params inject,[model=paramA],[dir=paramB],[bs=paramC],[count=paramD],[duration=paramE]
```
The parameters in brackets ([]) are optional. Take note of the following parameters:
- model (optional): the work mode. Valid values: read and write. Default value: read.
- dir (optional): the destination path. You can specify a path to test pressure on different disks. Default value: root directory.
- bs (optional): the block size. The format of the parameter is the same as the format of the bs parameter in the dd command. The bs parameter specifies the block size for each read or write operation. The bs parameter can affect the read and write speeds. The default block size is 1 MB, which helps maximize the I/O bandwidth of the disk.
- count (optional): the number of blocks in a read loop. Default value: 1024.
- duration (optional): the duration of fault injection. Unit: seconds. Default value: 300.
The following command output indicates that the ACS-ECS-HighIo plug-in is successfully run.
Run the iotop command to check whether the fault injection is performed as expected.
The following command output indicates that the fault injection is successful.

Fault recovery

Run the following command to resolve the high disk I/O load issue on the instance. Then, check whether the original read and write I/O speeds before fault injection recover.

Important

You must run this command after the drill is completed to delete the generated temporary file.

sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params recover

Example

Perform the fault injection by continuously reading a file in the usr path for 60 seconds.
```
sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params inject,dir=/usr,duration=60
```
The command output shown in the following figure indicates the fault injection parameters, the injection result, and the directory in which the temporary file is stored.
Check whether the AliFaultHighio.tmp temporary file is generated in the usr directory.
Run the iotop command to query the disk read and write I/O speeds.

Run the following command to resolve the high disk I/O load issue.

sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params recover