This topic describes how to troubleshoot and resolve high network bandwidth usage on Linux instances, including tool usage and optimization methods. - Simple Application Server

Symptoms

Service access issues: Service response time increases significantly, leading to request timeouts or service unavailability.
Monitoring alerts: You receive a text message or email alert that the network bandwidth usage has exceeded the preset alert threshold.

Causes

Malicious programs or processes: The instance is infected with a mining program or a Trojan, or is being used as part of a DDoS botnet. These malicious programs generate a large amount of abnormal network traffic.
Network attacks: Malicious attacks, such as a DDoS attack or brute-force attack, target the instance's public-facing ports and saturate the inbound bandwidth with invalid requests.
Insufficient instance network specifications: As your business grows, the instance's bandwidth specification can no longer handle the normal service traffic, creating a performance bottleneck.

Solutions

First, use the sar tool to locate the high-traffic network interface card (NIC). Then, use the iftop tool to identify the peer IP addresses consuming the bandwidth or the nethogs tool to identify high-traffic processes. Finally, take appropriate action based on the identified process and IP address.

Step 1: Identify the high-traffic network interface card

Use the sar tool to identify the high-traffic NIC and narrow the scope of your investigation.

Log on to the Simple Application Server using a rescue connection.
1. Go to the Servers page in the Simple Application Server console.
2. On the server card, click Remote Connection. In the Rescue Connection section of the dialog box, click Log On Now. Enter the username and password to log on to the Simple Application Server.

Gather network interface statistics.

# -n DEV: Reports network device statistics
# 1 5: Samples every 1 second, 5 times in total
sudo sar -n DEV 1 5

Identify the high-traffic NIC.
Focus on the txkB/s value in the Average section. Compare the values to find the IFACE (NIC name) with the highest value.
rxkB/s represents the average outbound bandwidth in kilobytes per second.
In the example, the eth0 NIC has the highest txkB/s value, which identifies it as the high-traffic NIC.

Step 2: Analyze and resolve the high-traffic issue

Analyze the NIC traffic.
- iftop: Monitors NIC traffic from a connection perspective. This tool helps you identify the IP addresses and ports with the highest traffic to and from your instance.
  For web services, use iftop to locate high-traffic IP addresses, then use tools like logwatch to analyze web logs and determine if the traffic is legitimate.
- nethogs: Monitors NIC traffic from a process perspective. This tool helps you identify the processes that are consuming the most network bandwidth.
iftop tool
1. Install the iftop tool.
  - For Alibaba Cloud Linux and CentOS:
    sudo yum install -y epel-release sudo yum install -y iftop
  - For Ubuntu and Debian:
    sudo apt update sudo apt install -y iftop
2. Monitor the high-traffic NIC.
  Replace <IFACE> with the high-traffic NIC name from Step 1.
```
# -i <IFACE>: Specifies the NIC to monitor as <IFACE>
# -P: Displays the port number (Port)
sudo iftop -i <IFACE> -P
```
  For example, if the high-traffic NIC is eth0, run sudo iftop -i eth0 -P.
3. Analyze the NIC traffic to find the peer IP address that consumes the most bandwidth.
  The real-time traffic information is sorted in descending order. The => symbol indicates the outbound data rate from your instance to a peer IP address. In the example, the average outbound traffic from the local instance to the IP address 140.205.11.x over the last 2 seconds is 4.32Mb/s.
4. Enter q to exit the iftop tool.
5. View the process associated with the port.
  Replace <HIGH_TRAFFIC_PEER_IP> with the peer IP address you found in the previous step.
```
sudo netstat -antp | grep <peer IP address that consumes bandwidth>
```
  Example output:
  In the example, the local IP address is 172.16.0.x, and the peer IP address is 140.205.11.x. The corresponding process is nginx: worker, with a process ID (PID) of 2282.
nethogs tool
Example
1. Install the nethogs tool.
  For Alibaba Cloud Linux and CentOS:
  sudo yum install -y epel-release sudo yum install -y nethogs
  For Ubuntu and Debian:
  sudo apt update sudo apt install -y nethogs
2. Monitor the high-traffic NIC.
  Replace <IFACE> with the high-traffic NIC name from Step 1.
  # The default monitoring interval is 1 second. You can use the -d parameter to specify the monitoring interval. sudo nethogs <IFACE>
  For example, if the high-traffic NIC is eth0, run sudo nethogs eth0.
3. Analyze the NIC traffic.
  The SENT column shows the rate at which your instance is sending data. In this example, the process consuming the most traffic is nginx: worker process, with an outbound traffic rate of about 696 KB/s and a process ID (PID) of 2282. Enter q to exit the tool.
Choose a solution based on the process or peer IP address.
- If the identified process (for example, a download tool such as wget or curl, or an unknown program) exhibits suspicious behavior, or if it is communicating with a malicious IP address:
  - Stop the abnormal process: You can end the abnormal process by running sudo kill -15 <PID>. Replace <PID> with the PID of the high-traffic process.
    Important
    Before you end the process, make sure it is not a core business process to avoid service interruptions.
  - Block the malicious IP address: Set up a firewall to block access from the malicious IP address.
  - Scan for and remove malicious programs: Use the virus scan feature (a paid feature) of Security Center to perform a full scan of the instance and remove any detected viruses.
- If the high traffic is generated by a normal business process:
  - Optimize the program: Check your business code for optimization opportunities, such as reducing unnecessary data transfers, adding caching, or compressing data.
  - Limit the rate: If your business allows, you can use tools such as iptables to limit the traffic rate for specific IP addresses or ports. This prevents a single user or service from consuming all the bandwidth.

Next steps

If you replace your server with an ECS instance, see Migrate data from a Simple Application Server to an ECS instance for instructions on data migration.
Periodically collect system metrics for continuous analysis. For more information, see Use the atop tool to monitor Linux system metrics.
You can purchase Anti-DDoS Origin or Anti-DDoS Pro and Anti-DDoS Premium and configure protected objects and mitigation policies to enhance security.