Test service performance during network packet loss in ECS - Elastic Compute Service

Network packet loss is a common network failure. Network congestion, hardware failures, link interference, and other situations can cause network packet loss. This drill scenario verifies system alerts and recovery when network packet loss occurs.

Limits

This drill scenario requires the tc Traffic Control (TC) utility and its dependent kernel Network Emulator (NetEm) component.

If the system lacks the tc utility, run the yum install -y iproute-tc or apt-get install -y iproute2 command to install it, or specify relevant parameters to automatically install it when you run the ACS-ECS-NetDelay plugin.
If the system kernel lacks the NetEm component, such as in CentOS, run the yum install kernel-modules-extra command to install the component package and restart the system.
Warning
Note that installing the kernel-modules-extra package changes the kernel version. Proceed with caution. We recommend that you do the drill using Elastic Compute Service (ECS) instances running other operating systems.

Implementation

This solution uses the Cloud Assistant plug-in ACS-ECS-NetLoss, which uses the tc utility and netem component to add specific rules to the network interface card, implementing TC for the Linux kernel. This plug-in can restrict either all IP addresses globally or target a single IP address. The Cloud Assistant network segment 100.100.0.0/16 will not be affected.

User guide

Prerequisites

Cloud Assistant Agent is installed on the ECS instance for which you want to perform a drill.
The status of Cloud Assistant is Normal on the ECS instance. For more information, see View the status of Cloud Assistant and handle anomalies.

Fault injection

Log on to the ECS instance as a user with sudo privileges.
For more information, see Use Workbench to connect to a Linux instance over SSH.
Run the Cloud Assistant plug-in ACS-ECS-NetLoss.
```
sudo acs-plugin-manager --exec --plugin ACS-ECS-NetLoss --params inject,dev=eth0,[rate=paramA],[target-ip=paramB],[replace=paramC],[duration=paramD],[install-tc=paramF]
```
Parameters in [] are optional fault injection parameters. See the following parameter descriptions:
- dev (required): The network interface card into which to inject faults, such as eth0. You can use ifconfig to view the current system's network interface cards.
- rate (optional): Packet loss percentage. The default is 10 (%).
- target-ip (optional): The target IP address expected to be affected. The default is empty, indicating packet loss rules will be added for all IP addresses. If a target IP address is specified, only that IP address will be affected.
- replace (optional): If the network interface card already has configured TC rules, injection will fail due to conflicts. If you want to overwrite the existing rules, you can set this value to true.
- duration (optional): Duration in seconds. The default is 300.
- install-tc (optional): If the system lacks the tc utility, you can set this parameter to true for automatic installation. The default is false.
  The following command output indicates that the Cloud Assistant plug-in ACS-ECS-NetLoss is run successfully.
Check if the fault injection was successful.
Ping the target network to check the network packet loss situation. As shown in the figure below, you can see that network packet loss is occurring, with an average packet loss of approximately 44%.

Fault recovery

Method 1: Wait for automatic recovery after timeout.

Method 2: Run the following fault recovery command:

sudo acs-plugin-manager --exec --plugin ACS-ECS-NetLoss --params recover