:Query and Case Analysis of Linux Network Traffic Load

Disclaimer: This topic may contain information about third-party products. The information is for reference only. Alibaba Cloud does not make a guarantee in any form of the performance and reliability of the third-party products, and potential impacts of operations on these products.

Overview

Linux has various tools for viewing network traffic, such as sar, iftop, and nethogs. They can analyze the flow information in the system from different latitudes, which is briefly introduced in this article.

Description

Take note of the following items:

• Before you perform high-risk operations such as modifying the specifications or data of an Alibaba Cloud instance, we recommend that you check the disaster recovery and fault tolerance capabilities of the instance to ensure data security.
• Before you modify the specifications or data of an Alibaba Cloud instance, such as an Elastic Compute Service (ECS) instance or an ApsaraDB RDS instance, we recommend that you create snapshots or enable backups for the instance. For example, you can enable log backups for an ApsaraDB RDS instance.
• If you have granted specific users the permissions on sensitive information, such as usernames and passwords, or submitted sensitive information in the Alibaba Cloud Management Console, we recommend that you modify the sensitive information at the earliest opportunity.

For the problem of full or high bandwidth caused by normal processes, you need to upgrade the bandwidth of the server. For abnormal processes, it may be caused by malicious program problems or malicious access to some IP addresses, or the service may be attacked by CC. You can perform the following steps to troubleshoot and locate the problem:

1. Locate the problem. Find the specific process that affects the excessive bandwidth usage.

2. Analysis and processing. Check whether the processes that affect the excessive bandwidth usage are normal, and classify them for processing.

   • For normal processes, you must optimize or upgrade the bandwidth.
   • For abnormal processes: You can manually detect processes or use third-party security tools.

The configurations and descriptions in this topic have been tested in CentOS 6.5 64-bit operating systems. The configurations of other types and versions of operating systems may vary. For more information, see the official documentation of the corresponding operating system.

Use of sar

sar can analyze the sending and sending of data packets and error messages from the network interface level. The following common commands are used to monitor network traffic using sar.

sar -n DEV [interval] [count]

Note: The interval parameter specifies the statistical interval. The count parameter specifies the number of times.

Example

Run the following command to use sar to count the activity status of the network interface every 1 second and display it five times in a row.

sar -n DEV 1 5

command execution after each network interface controller 5 times of the average data value, according to the actual situation to determine the bandwidth run full network interface controller name for ECS example, usually must not use the intranet network interface controller for eth0, internet network interface controller for eth1. The system display is similar to the following. Confirm that eth1 passes through a large amount of traffic, and then perform the following steps to check the specific problem.

Description of the main fields of the display result

• IFACE: the name of the network interface.
• rxpck/s and txpck/s: the number of packets received or sent per second.
• rxkB/s and txkB/s: the number of bytes received or sent per second, in kB/s.
• rxcmp/s and txcmp/s: the number of compressed packets received or sent per second.
• rxmcst/s: multicast packets received per second.

Use of iftop

Common usage of the iftop command is as follows. The interface followed by parameter-i indicates the network interface name, such as eth0 and eth1. If the interface name is not specified by the-i parameter, the usage of the first network interface controller is detected by default.

iftop [-i interface]

Example

1. Log on to a Linux instance. For more information about how to log on to a Linux instance, see Connect to a Linux instance by using a management terminal.
2. Run the following command to install the iftop traffic monitoring tool:

   Note: If the installation fails, you can switch the source of CentOS 6. For more information, see How to switch the source of CentOS 6 EOL.

   yum install iftop -y

3. Run the following command to check which port has high traffic and the internal network traffic:

   iftop -i eth1 -P

   Note: The-P parameter displays the request port.

   A command output similar to the following one is returned.
   
   
4. Run the following command to view the process corresponding to the port:

   netstat -tunlp grep [$Port]

   Note :[$Port] indicates the port that you viewed in the previous step.

   A command output similar to the following one is returned.
   
   
5. After confirming the corresponding service, you can stop the service or use the iptables service to process the specified address, such as blocking IP addresses or limiting the speed of the server to ensure that the server bandwidth can be used normally.

Description of the main fields of the display result

• First line: Bandwidth usage is displayed.
• The middle part is a list of external connections, which records which IP addresses are connecting to the local network.
• The right part of the middle part is the real-time traffic information, which is the average traffic of 2 seconds, 10 seconds and 40 seconds respectively when the access IP is connected to this machine.
• => represents sending data and <= represents receiving data.
• The bottom three rows.
  • First column: TX indicates the sent traffic, RX indicates the received traffic, and TOTAL indicates the total traffic.
  • cum in the second column: indicates the total traffic for each situation in the first column.
  • Third column peak: indicates the peak traffic in each case in the first column.
  • rate in the fourth column: indicates the average traffic within 2 seconds, 10 seconds, and 40 seconds for each condition in the first column.

Note: The traffic unit of iftop is Mb, where b is bit, not byte. The unit displayed by ifstat is B in KB, which is byte. 1byte equals 8 bits.

Description of common operation commands

After entering the iftop interface, you can adjust the display results by pressing the corresponding letter shortcut button.

• h: specifies whether to display help.
• n: toggles to display the IP address or host name of the local machine.
• s: specifies whether to display the host information of the local machine.
• d: switches whether to display the host information of the remote target host.
• t: Switch the display format to 2 lines, 1 line, only the sent traffic, or only the received traffic.
• N: Switch to display the port number or port service name.
• S: switches whether to display the port information of the local machine.
• D: switches whether to display the port information of the remote target host.
• p: specifies whether to display port information.
• P: Switch to pause or continue display.
• b: toggles whether to display the average traffic graph bar.
• B: Switch over to calculate the average traffic within 2 seconds or 10 seconds or 40 seconds.
• T: toggles whether to display the total traffic of each connection.
• L: Turn on the screen filtering function and enter the characters to be filtered. For example, after entering the corresponding IP address and entering, the screen will only display the traffic information related to this IP address.
• L: Switch the scale on the top of the display screen. If the scale is different, the flow graph bar will change.
• J or k: Scroll up or down the connection record displayed on the screen.
• 1 or 2 or 3: Sort the traffic data according to the three columns displayed on the right.
• <: Sort by the local name or IP address on the left.
• >: Sort according to the host name or IP address of the remote target host.
• o: specifies whether to display only the current connection.
• f: edit the filter code.
• ! : calls shell commands.
• q: exits.

Use of Nethogs

Nethogs is an open-source network traffic monitoring tool that can be used to display the bandwidth usage of each process. This allows you to more intuitively locate the source of abnormal traffic. Nethogs supports IPv4 and IPv6 protocols, and supports local network interface controller and PPP connections. For more information about how to install the Nethogs tool, see its official help documentation. After the installation is completed, directly enter the nethogs startup tool. If you do not have any parameters, nethogs monitors eth0 by default. You can use commands such as ifconfig to verify which network interface (eth1, eth0) corresponds to the public network interface controller.

Example

1. Log on to a Linux instance. For more information about how to log on to a Linux instance, see Connect to a Linux instance by using a management terminal.
2. Run the following command to install the nethogs traffic monitoring tool:
   

   Note: If the installation fails, you can switch the source of CentOS 6. For more information, see How to switch the source of CentOS 6 EOL.

   yum install nethogs -y

   
   
3. Use the nethogs tool to view the process-level traffic information on network interface controller. If the traffic of the eth1 network interface controller is high, run the following command to view the network bandwidth of each process and the PID corresponding to the process, and determine the specific process that causes the bandwidth to run full or high.

   nethogs eth1

   Note: The default monitoring interval for nethogs is 1 second. You can set the monitoring interval by using the-d parameter. For example, run the following command to set the monitoring interval to 5 seconds.
   nethogs eth1 -d 5
   

   If an output similar to the following one is displayed, Python is installed.
   
   
4. If the process determines that it is a malicious program, you can terminate the process by executing the following command.

   kill -TERM [$Port1]

   Note: If you are using a Web service program, you can use tools such as iftop to query the source of specific IP addresses and then analyze whether the Web access logs are normal traffic. log analysis can be performed by using tools such as logwatch or awstats.

Description of the display result

On the nethogs monitoring page, you can sort by s in the Sent column, by r in the Received column, and by m in different statistical units, such as kb/s, kb, b, and mb. The following table describes the monitoring interface information.

• The PID column indicates the process number of the corresponding traffic association program.
• The USER column indicates the user to which the corresponding process belongs.
• The PROGRAM column indicates the specific execution path of the program.
• The name of the network interface currently monitored in the DEV column.
• The Sent column indicates the data traffic that has been sent by the corresponding process.
• The Received column indicates the data traffic that the program has received.

Applicable scope

• ECS