Symptoms
-
Service access issues: Service response time increases significantly, causing request timeouts or service inaccessibility.
-
High monitoring metrics: In the ECS console or CloudMonitor, your instance's outbound Internet bandwidth usage consistently exceeds 80% and approaches or reaches the bandwidth limit. You may also notice a sudden increase in the number of network connections.
-
Monitoring alerts: You receive an SMS message or email notifying you that the network bandwidth usage has exceeded the configured alert threshold.
Causes
-
Abnormal processes or malicious programs: The instance is infected with cryptocurrency mining programs or Trojans, or it is acting as a bot in a DDoS attack. These malicious programs generate a large amount of abnormal network traffic.
-
Network attacks: The public-facing ports of the instance are targeted by malicious access attempts, such as application-layer CC attacks, DDoS attacks, or brute-force attacks. These attacks can saturate the inbound bandwidth with invalid requests.
-
Insufficient instance network capacity: As your business grows, the instance's configured bandwidth can no longer handle the normal service traffic, creating a network performance bottleneck.
Solution
First, use the sar tool to locate the network interface with abnormal traffic. Then, analyze the NIC traffic by using the iftop tool to identify the peer IP address consuming the bandwidth, or the nethogs tool to identify the high-traffic process. Finally, take appropriate actions based on the nature of the process and IP address.
Step 1: Locate the high-traffic NIC
Use the sar tool to identify the high-traffic NIC and define the troubleshooting target.
Log on to an ECS instance using a VNC connection.
Go to ECS console - Instances. In the top-left corner, select the region and resource group for the target instance.
Navigate to the details page of the target instance. Click Connect and select VNC. Enter your credentials to log on to the ECS instance.
-
Gather network interface statistics.
# -n DEV: Reports network device statistics # 1 5: Samples every 1 second for a total of 5 times sudo sar -n DEV 1 5 -
Identify the high-traffic NIC.
Focus on the
Averagevalue oftxkB/s(average outbound bandwidth). By comparing the values, find theIFACE(NIC name) with the highest value.rxkB/srepresents the average inbound bandwidth rate.Linux 5.10.134-19.1.al8.x86_64 (iZbp1e5xxx) 11/14/2025 _x86_64_ (2 CPU) 04:11:13 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 04:11:14 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 04:11:14 PM eth0 317.00 580.00 21.36 933.00 0.00 0.00 0.00 0.00 04:11:14 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 04:11:14 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 04:11:15 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 04:11:15 PM eth0 328.00 582.00 22.34 943.08 0.00 0.00 0.00 0.00 04:11:15 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 04:11:15 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 04:11:16 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 04:11:16 PM eth0 352.00 595.00 24.08 935.58 0.00 0.00 0.00 0.00 04:11:16 PM eth1 1.00 1.00 0.04 0.04 0.00 0.00 0.00 0.00 04:11:16 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 04:11:17 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 04:11:17 PM eth0 347.00 581.00 23.58 943.02 0.00 0.00 0.00 0.00 04:11:17 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 04:11:17 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 04:11:18 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 04:11:18 PM eth0 367.00 623.00 24.98 952.79 0.00 0.00 0.00 0.00 04:11:18 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: eth0 342.20 592.20 23.27 941.49 0.00 0.00 0.00 0.00 Average: eth1 0.20 0.20 0.01 0.01 0.00 0.00 0.00 0.00In this example, the
eth0NIC has the highesttxkB/svalue. Therefore,eth0is the high-traffic NIC.
Step 2: Analyze and resolve the high-traffic issue
-
Analyze the traffic on the NIC.
-
iftop: Monitors NIC traffic from a connection perspective. It identifies the IP addresses and ports that generate the most traffic with your instance.For web services, you can use
iftopto locate high-traffic IP addresses and combine it with tools likelogwatchto analyze web logs and determine if the traffic is legitimate. -
nethogs: Monitors NIC traffic from a process perspective. It identifies the process that consumes the most network bandwidth.
iftop
-
Install the
iftoptool.-
For Alibaba Cloud Linux and CentOS:
sudo yum install -y iftop -
For Ubuntu and Debian:
sudo apt update sudo apt install -y iftop
-
-
Monitor the high-traffic NIC.
Replace
<IFACE>with the name of the high-traffic NIC that you identified in Step 1.# -i <IFACE>: Specifies the NIC to monitor. # -P: Displays port numbers. sudo iftop -i <IFACE> -PFor example, if the high-traffic NIC is
eth0, runsudo iftop -i eth0 -P. -
Analyze the NIC traffic to find the peer IP address consuming the bandwidth.
19.1Mb 38.1Mb 57.2Mb 76.3Mb 95.4Mb iZbp1e5w04k4l xxx => 140.205.11 xxx 4.32Mb 3.17Mb 1.06Mb <= 62.9Kb 47.7Kb 15.1Kb iZbp1e5w04k4l xxx => 140.xxx 0b 164Kb 41.1Kb <= 0b 4.97Kb 1.30Kb iZbp1e5w04k4l xxx => 140.xxx 796Kb 159Kb 40.1Kb <= 15.5Kb 3.10Kb 804b iZbp1e5w04k4l xxx => 140.xxx 0b 89.9Kb 29.2Kb <= 0b 2.97Kb 882b iZbp1e5w04k4l xxx => 140.xxx 0b 83.9Kb 46.4Kb <= 0b 2.55Kb 1.25Kb iZbp1e5w04k4l xxx => 140.xxx 0b 78.9Kb 50.7Kb <= 0b 2.50Kb 1.52Kb iZbp1e5w04k4l xxx => 140.xxx 0b 71.0Kb 89.6Kb <= 0b 2.51Kb 2.85Kb iZbp1e5w04k4l xxx => 140.xxx 0b 66.1Kb 26.2Kb <= 0b 2.29Kb 822b iZbp1e5w04k4lv xxx => 140.xxx 0b 65.6Kb 16.4Kb <= 0b 1.62Kb 416b iZbp1e5w04k4l xxx => 140.xxx 0b 61.1Kb 23.2Kb <= 0b 1.77Kb 656b iZbp1e5w04k4l xxx => 140.xxx 0b 56.2Kb 55.6Kb <= 0b 1.53Kb 1.54Kb iZbp1e5w04k4lv xxx => 140.xxx 0b 52.1Kb 75.3Kb <= 0b 1.65Kb 2.40Kb iZbp1e5w04k4l xxx => 140.xxx 0b 52.0Kb 13.0Kb <= 0b 1.42Kb 363b ───────────────────────────────────────────────────────────────────────────────────────────────────── TX: cum: 80.9MB peak: 14.3Mb rates: 7.15Mb 6.52Mb 11.2Mb RX: 1.91MB 440Kb 146Kb 154Kb 276Kb TOTAL: 82.8MB 14.6Mb 7.30Mb 6.67Mb 11.4MbThe output sorts real-time traffic information in descending order. The
=>symbol indicates the rate at which your instance is sending data to a peer IP address. In this example, the average outbound traffic rate from the instance to the IP address140.205.11.xover the last 2 seconds is4.32Mb/s. -
Press
qto exit theiftopinterface. -
View the process that corresponds to the port.
Replace
<PEER_IP_ADDRESS>with the peer IP address that you obtained in the previous step.sudo netstat -antp | grep <PEER_IP_ADDRESS>The following example shows a sample output.
tcp 0 172.16.0.xxx xxx 140.205.11.xxx xxx ESTABLISHED 2282/nginx: worker tcp 0 172.16.0.xxx xxx 140.205.11.xxx xxx ESTABLISHED 2282/nginx: worker tcp 0 172.16.0.xxx xxx 140.205.11.xxx xxx ESTABLISHED 2282/nginx: worker tcp 0 172.16.0.xxx xxx 140.205.11.xxx xxx ESTABLISHED 2282/nginx: worker tcp 0 172.16.0.xxx xxx 140.205.11.xxx xxx ESTABLISHED 2282/nginx: worker tcp 0 172.16.0.xxx xxx 140.205.11.xxx xxx ESTABLISHED 2282/nginx: workerIn this example, the local IP address is
172.16.0.x, the peer IP address is140.205.11.x, the corresponding process isnginx: worker, and the PID is2282.
nethogs
Example
-
Install the nethogs tool.
-
For Alibaba Cloud Linux and CentOS:
sudo yum install -y nethogs -
For Ubuntu and Debian:
sudo apt update sudo apt install -y nethogs
-
-
Monitor the high-traffic NIC.
Replace
<IFACE>with the name of the high-traffic NIC that you identified in Step 1.# The default monitoring interval is 1 second. You can use -d to specify the monitoring interval. sudo nethogs <IFACE>For example, if the high-traffic NIC is
eth0, runsudo nethogs eth0. -
Analyze the NIC traffic.
NetHogs version 0.8.5 PID USER PROGRAM DEV SENT RECEIVED 2282 nginx nginx: worker process 696.898 19.289 KB/sec 15823 root sshd: root@pts/3 0.773 0.231 KB/sec ? root 172.16.xxx.54.137:80 0.000 0.000 KB/sec ? root 172.16.xxx.26.209:80 0.000 0.000 KB/sec ? root 172.16.xxx.45.106:80 0.000 0.000 KB/sec ? root 172.16.xxx.4.191:80 0.000 0.000 KB/sec ? root 172.16.xxx.154.78:80 0.000 0.000 KB/sec 1749 root /usr/local/cloudmonitor/bin/argusagent 0.000 0.000 KB/sec ? root 172.16.xxx.54.135:80 0.000 0.000 KB/sec ? root 172.16.xxx.26.207:80 0.000 0.000 KB/sec ? root 172.16.xxx.4.217:80 0.000 0.000 KB/sec ? root 172.16.xxx.89.115:80 0.000 0.000 KB/sec 6215 root /opt/aliyun-security/ilogtail/ilogtail-aliyun-security 0.000 0.000 KB/sec ? root 172.16.xxx.154.78:80 0.000 0.000 KB/sec ? root 172.16.xxx.83.69:80 0.000 0.000 KB/sec ? root 172.16.xxx.54.132:80 0.000 0.000 KB/sec ? root 172.16.xxx.26.210:80 0.000 0.000 KB/sec ? root 172.16.xxx.4.217:80 0.000 0.000 KB/sec ? root 172.16.xxx.89.113:80 0.000 0.000 KB/sec ? root 172.16.xxx.154.78:80 0.000 0.000 KB/sec 1806 root /usr/local/aegis/aegis_client/aegis_12_61/AliYunDun 0.000 0.000 KB/sec 2810 root /usr/local/aegis/alihips/AliHips 0.000 0.000 KB/sec 6172 root /opt/aliyun-security/rapt-daemon/rapt-daemon 0.000 0.000 KB/sec ? root unknown TCP 0.000 0.000 KB/secThe
SENTcolumn shows the rate at which your instance is sending data to the receiver (the peer IP address). In this example, the process consuming the most traffic isnginx: worker process, with an outbound traffic rate of about 696 KB/s and a process ID (PID) of2282. Pressqto exit the tool's interface.
-
-
Choose a solution based on the identified process or peer IP address.
-
If the identified process (such as a download tool like
wgetorcurl, or an unknown program) acts suspiciously, or if it is communicating with a malicious peer IP address:-
Terminate the abnormal process: Run the
sudo kill -15 <PID>command. Replace<PID>with the PID of the identified high-traffic process.ImportantBefore terminating a process, ensure it is not a critical business process to avoid service disruptions.
-
Block the malicious IP address: Use security group rules to block known malicious IP addresses. If you are experiencing an application-layer CC attack where the source IP addresses are widely distributed, security group rules are less effective. In this case, we recommend that you deploy a web application firewall (WAF) for specialized protection. For more information, see Add an ECS instance to WAF to defend against CC attacks.
-
Scan for and remove malicious programs: Enable the virus detection and removal feature in Security Center to perform a full scan of the instance and remove any detected malicious programs.
-
Respond to high-frequency DDoS attacks: During high-frequency DDoS or CC attacks, peak traffic may exceed the blackhole threshold of Anti-DDoS Origin. This can cause your cloud service to enter a blackhole state and become inaccessible. In this scenario, you must enable Anti-DDoS Pro and Anti-DDoS Premium to ensure service availability.
-
-
If a legitimate business process generates the high traffic, this is likely due to normal business demand:
-
Upgrade bandwidth: The instance's current network capacity is a bottleneck. You should upgrade the instance bandwidth.
-
Optimize the application: Review your application code for potential optimizations, such as reducing unnecessary data transfers, adding caching, or compressing data.
-
Limit the traffic rate: If your service allows, you can use tools like
iptablesto limit the traffic rate for specific IP addresses or ports. This prevents a single user or service from consuming all the available bandwidth.
-
-
If you cannot find any processes consuming abnormal bandwidth but overall usage remains high, the total service traffic has likely exceeded the instance's network capacity. In this case, you should upgrade the instance bandwidth.
-
Next steps
-
We recommend that you periodically collect system metrics for ongoing analysis. For more information, see Use the atop tool to monitor Linux system metrics.
-
Configure instance monitoring and alerting to receive notifications about future risks and anomalies.