Packet forwarding rate, network bandwidth, and network latency are important metrics to measure the network performance of Elastic Compute Service (ECS) instances. Testing network performance can help increase network stability and throughput, reduce network latency, and improve user experience and application performance. This topic describes how to test the packet forwarding rate, network bandwidth, and network latency of an ECS instance. You can use the test result to estimate the network performance of the instance. In this topic, Netperf and sockperf are used to test the network performance of ECS instances.
Prerequisites
ECS instances that meet the requirements for the test environment are created. For information about how to create an ECS instance, see Create an instance by using the wizard.
In the same test environment, all ECS instances must be deployed in the same virtual private cloud (VPC), be connected to the same vSwitch, and belong to the same security group.
Prepare the environment
To prevent data loss, we recommend that you use a tool to test the network performance of new ECS instances that contain no data.
Instance type specifications are all verified and obtained within a test environment. In actual scenarios, the performance of an instance may vary based on other factors such as instance load and networking model.
In this test, ECS instances whose network performance is tested are used as test machines and auxiliary test machines. The packet forwarding rate, network bandwidth, and network latency of these ECS instances are tested. Test machines can be used as clients or servers in tests that use Netperf or sockperf. Auxiliary test machines can also be used as clients or servers in tests that use Netperf or sockperf. Test machines are connected to auxiliary test machines to transmit test configurations.
The following items describe the test items and recommended quantities of test machines in different test environments:
Test the packet forwarding rate of an ECS instance
Less than 6 million pps
Test item
Test machine
Auxiliary test machine
Instance type
ecs.g7.large
ecs.g7.large
Image
Alibaba Cloud Linux 3
Alibaba Cloud Linux 3
Quantity
1
1
Greater than 6 million pps and less than 20 million pps
Test item
Test machine
Auxiliary test machine
Instance type
ecs.g7.16xlarge
ecs.g7.16xlarge
Image
Alibaba Cloud Linux 3
Alibaba Cloud Linux 3
Quantity
1
3
Greater than 20 million pps
Test item
Test machine
Auxiliary test machine
Instance type
ecs.g7.32xlarge
ecs.g7.32xlarge
Image
Alibaba Cloud Linux 3
Alibaba Cloud Linux 3
Quantity
1
3
Test the network bandwidth and network latency of an ECS instance
NotePacket forwarding rate is not a factor to consider when you test the network bandwidth and network latency of an ECS instance. You can select any instance type for the test. In this test, the instance type described in the following table is used.
Test item
Test machine
Auxiliary test machine
Instance type
ecs.g7.large
ecs.g7.large
Image
Alibaba Cloud Linux 3
Alibaba Cloud Linux 3
Quantity
1
1
Test the packet forwarding rate
Less than 6 million pps
Connect to the test machine and the auxiliary test machine.
For information about the methods for connecting to an instance, see Connection method overview.
Run the following command on the test machine and the auxiliary test machine to download Netperf:
wget https://benchmark-packages.oss-cn-qingdao.aliyuncs.com/netperf-2.7.0.tar.gz
Run the following commands on the test machine and the auxiliary test machine to install Netperf and SAR.
Run the following command to decompress the Netperf installation package:
sudo yum install -y gcc autoconf automake libtool sysstat tar -zxvf netperf-2.7.0.tar.gz
Run the following command to query the version number of
GCC
:gcc -v 2>&1
If the version of
GCC
on the test machine and the auxiliary test machine are later than V10, you must perform the following operations to ensure thatGCC
can be compiled properly. Otherwise, ignore this step.Run the following command to modify the
nettest_omni.c
file:cd netperf vim src/nettest_omni.c
Press the i key to enter the edit mode and manually delete the following declared variables in the
nettest_omni.c
file:/* different options for the sockets */ int loc_nodelay, /* don't/do use NODELAY locally */ rem_nodelay, /* don't/do use NODELAY remotely */ loc_sndavoid, /* avoid send copies locally */ loc_rcvavoid, /* avoid recv copies locally */ rem_sndavoid, /* avoid send copies remotely */ rem_rcvavoid; /* avoid recv_copies remotely */
After you delete the previous variables, press the Esc key. Then, enter
:wq
and press the Enter key to save the modification and exit.
Run the following command to install Netperf and SAR:
cd netperf sudo ./configure sudo make && sudo make install
Run the following command on the test machine to start 64 netserver services:
#!/bin/bash for j in `seq 64`; do netserver -p $[16000+j] > server_$[16000+j].netperf > /dev/null 2>&1 & done
Run the following command on the test machine to query the private IP address of the test machine:
ifconfig || ip addr
Run the following command on the auxiliary test machine to send traffic to the test machine:
#!/bin/bash server_ip=<Private IP address of the test machine> for j in `seq 64`; do port=$[16000+j] netperf -H ${server_ip} -l ${run_time:-300} -t UDP_STREAM -p $port -- -m 1 -D > /dev/null 2>&1 & done
Replace
<Private IP address of the test machine>
with the private IP address of the test machine obtained in the previous step, as shown in the following figure.Run the following command on the test machine to test the network traffic:
sar -n DEV 1
View the values in the
rxpck/s
column in the test result. Therxpck/s
column shows the total number of packets received by the test machine per second. In this example, the average number of packets received by the test machine per second is approximately 940,000, as shown in the following figure.
Greater than 6 million pps and less than 20 million pps
Connect to the test machine and the auxiliary test machine.
For information about the methods for connecting to an instance, see Connection method overview.
Run the following command on three auxiliary test machines and one test machine to install sockperf:
sudo yum install -y autoconf automake libtool gcc-c++ cd /opt sudo wget https://github.com/Mellanox/sockperf/archive/refs/tags/3.8.tar.gz sudo tar -zxf 3.8.tar.gz cd sockperf-3.8/ sudo ./autogen.sh sudo ./configure sudo make -j `cat /proc/cpuinfo| grep process | wc -l` sudo make install
Run the following command on the test machine to query the private IP address of the test machine:
ifconfig || ip addr
Run the following command on three auxiliary test machines to send traffic to the test machine:
server_ip="<Private IP address of the test machine>" threads=64 msg_size=14 run_time=60 basePort=6666 for((i=0;i<$threads;++i));do nohup sockperf tp -i $server_ip --pps max -m ${msg_size} -t ${run_time} --port $[${basePort}+${i}] 2>&1 & done
Replace
<Private IP address of the test machine>
with the private IP address of the actual test machine.run_time
specifies the time period during which traffic is sent.Run the following command on three auxiliary test machines and one test machine to test the network traffic:
sar -n DEV 1
View the values in the
rxpck/s
column on the test machine. Therxpck/s
column shows the total number of packets received by the test machine per second. In this example, the number of packets received by the test machine per second is approximately 12,000,000, as shown in the following figure.
Greater than 20 million pps
Connect to the test machine and the auxiliary test machine.
For information about the methods for connecting to an instance, see Connection method overview.
Run the following command on three auxiliary test machines and one test machine to install sockperf:
sudo yum install -y autoconf automake libtool gcc-c++ cd /opt sudo wget https://github.com/Mellanox/sockperf/archive/refs/tags/3.8.tar.gz sudo tar -zxf 3.8.tar.gz cd sockperf-3.8/ sudo ./autogen.sh sudo ./configure sudo make -j `cat /proc/cpuinfo| grep process | wc -l` sudo make install
Run the following command on the test machine to bind the interrupts:
a=$(cat /proc/interrupts | grep virtio2-input | awk -F ':' '{print $1}') cpu=0 for irq in $a; do echo $cpu >/proc/irq/$irq/smp_affinity_list let cpu+=2 done
Run the following command on the test machine to query the private IP address of the test machine:
ifconfig || ip addr
Run the following command on three auxiliary test machines to send traffic to the test machine:
server_ip="<Private IP address of the test machine>" threads=64 msg_size=14 run_time=60 basePort=6666 for((i=0;i<$threads;++i));do nohup sockperf tp -i $server_ip --pps max -m ${msg_size} -t ${run_time} --port $[${basePort}+${i}] 2>&1 & done
Replace
<Private IP address of the test machine>
with the private IP address of the actual test machine.run_time
specifies the time period during which traffic is sent.Run the following command on three auxiliary test machines and one test machine to test the network traffic:
sar -n DEV 1
View the values in the
rxpck/s
column on the test machine. Therxpck/s
column shows the total number of packets received by the test machine per second. In this example, the number of packets received by the test machine per second is approximately 20,000,000, as shown in the following figure.
Test the network bandwidth
Connect to the test machine and the auxiliary test machine.
For information about the methods for connecting to an instance, see Connection method overview.
Run the following command on the test machine and the auxiliary test machine to download Netperf:
wget https://benchmark-packages.oss-cn-qingdao.aliyuncs.com/netperf-2.7.0.tar.gz
Run the following commands on the test machine and the auxiliary test machine to install Netperf and SAR.
Run the following command to decompress the Netperf installation package:
sudo yum install -y gcc autoconf automake libtool sysstat tar -zxvf netperf-2.7.0.tar.gz
Run the following command to query the version number of
GCC
:gcc -v 2>&1
If the version of
GCC
on the test machine and the auxiliary test machine are later than V10, you must perform the following operations to ensure thatGCC
can be compiled properly. Otherwise, ignore this step.Run the following command to modify the
nettest_omni.c
file:cd netperf vim src/nettest_omni.c
Press the i key to enter the edit mode and manually delete the following declared variables in the
nettest_omni.c
file:/* different options for the sockets */ int loc_nodelay, /* don't/do use NODELAY locally */ rem_nodelay, /* don't/do use NODELAY remotely */ loc_sndavoid, /* avoid send copies locally */ loc_rcvavoid, /* avoid recv copies locally */ rem_sndavoid, /* avoid send copies remotely */ rem_rcvavoid; /* avoid recv_copies remotely */
After you delete the previous variables, press the Esc key. Then, enter
:wq
and press the Enter key to save the modification and exit.
Run the following command to install Netperf and SAR:
cd netperf sudo ./configure sudo make && sudo make install
Run the following command on the test machine to start 64 netserver services:
#!/bin/bash for j in `seq 64`; do netserver -p $[16000+j] > server_$[16000+j].netperf > /dev/null 2>&1 & done
Run the following command on the test machine to query the private IP address of the test machine:
ifconfig || ip addr
Run the following command on the auxiliary test machine to send traffic to the test machine:
#!/bin/bash server_ip=<Private IP address of the test machine> for j in `seq 64`; do port=$[16000+j] netperf -H ${server_ip} -l ${run_time:-300} -t TCP_STREAM -p $port -- -D > /dev/null 2>&1 & done
Replace
<Private IP address of the test machine>
with the private IP address of the test machine obtained in the previous step, as shown in the following figure.Run the following command on the test machine to test the network bandwidth:
sar -n DEV 1
View the values in the
rxkB/s
column in the test result. TherxkB/s
column shows the size of packets received by the test machine per second. Unit: KB. The packet size and bandwidth can be converted to each other by using the following formula: Bandwidth (Kbit/s) = Packet size (KB) × 8.The following figure shows an example.
Test the network latency
Connect to the test machine and the auxiliary test machine.
For information about the methods for connecting to an instance, see Connection method overview.
Run the following commands on the test machine and the auxiliary test machine to install
sockperf
:sudo yum install -y autoconf automake libtool gcc-c++ cd /opt sudo wget https://github.com/Mellanox/sockperf/archive/refs/tags/3.8.tar.gz sudo tar -zxf 3.8.tar.gz cd sockperf-3.8/ sudo ./autogen.sh sudo ./configure sudo make -j `cat /proc/cpuinfo| grep process | wc -l` sudo make install
Run the following command on the test machine to query the private IP address of the test machine:
ifconfig || ip addr
Run the following command on the test machine to start sockperf:
sockperf sr --tcp --daemonize
Run the following command on the auxiliary test machine to send traffic to the test machine:
cd sockperf pp -i <Private IP address of the test machine> --tcp -t 30 -m 14 --full-log=result.json
Replace
<Private IP address of the test machine>
with the private IP address of the test machine obtained in the previous step.View the returned data, as shown in the following figure.
Returned data starting with
avg-latency
indicates the average latency in microseconds.Returned data starting with
percentile 99.000
indicates the 99-percentile latency in microseconds.