Test service performance when disk capacity utilization in ECS is high by performing a high disk capacity utilization drill - Elastic Compute Service

Excessively high disk capacity utilization may cause system performance degradation, system crashes, and data loss due to data accumulation and temporary file buildup. This topic describes how to perform a drill to verify that the system can run stably and does not encounter data loss if disk capacity utilization is high or even full on an Elastic Compute Service (ECS) instance.

Implementation

The Cloud Assistant plug-in named ACS-ECS-FillDisk is used. You can enter the destination path and disk capacity utilization. The plug-in generates a temporary file of the specified size by using fallocate to make the disk reach the specified high disk capacity utilization. You must manually delete the temporary file for recovery.

Procedure

Prerequisites

Cloud Assistant Agent is installed on the ECS instance for which you want to perform a drill.
The status of Cloud Assistant is Normal on the ECS instance. For more information, see View the status of Cloud Assistant and handle anomalies.

Fault injection

Log on to the ECS instance.
For more information, see Use Workbench to connect to a Linux instance over SSH.
Use a sudo user to run the ACS-ECS-FillDisk plug-in.
```
sudo acs-plugin-manager --exec --plugin ACS-ECS-FillDisk --params inject,[dir=paramA],[percent=paramB]
```
The parameters in brackets ([]) are optional. Take note of the following parameters:
- dir (optional): the disk directory. Default value: /.
- percent (optional): the destination disk capacity utilization. Default value: 80%.
Note
If the injection fails during the fallocate process because of insufficient space or other reasons, the generated temporary file is deleted.
The following command output indicates that the ACS-ECS-FillDisk plug-in is successfully run.
Run the df {dir} command to check whether the fault injection is successful.
The following command output indicates that the fault injection is successful.

Fault recovery

Log on to the ECS instance and run the rm -f {dir}/AliFaultFillDisk.tmp command to delete the temporary file. Replace dir with the path of the temporary file specified during injection.

Solutions to insufficient disk capacity

Resize disks whose capacity is insufficient.
In Container Service for Kubernetes (ACK) scenarios, automatically extend disk volumes if the utilization of the volumes exceeds a threshold. For more information, see Configure an automatic disk expansion policy to achieve automatic expansion.