Elastic Compute Service:Best practices for testing network performance

Prepare the environment

Test the packet forwarding rate (less than 6 million pps)

1. Connect to the test machine and the auxiliary test machine.
   For more information about the methods for connecting to an instance, see Connection methods.
2. Run the following command on the test machine and the auxiliary test machine to download Netperf:

   wget https://benchmark-packages.oss-cn-qingdao.aliyuncs.com/netperf-2.7.0.tar.gz

3. Run the following command on the test machine and the auxiliary test machine to install Netperf and SAR.
   1. Run the following command to decompress the Netperf installation package:

      yum install -y gcc autoconf automake libtool sysstat
      tar -zxvf netperf-2.7.0.tar.gz

   2. Run the following command to query the version number of GCC:

      gcc -v 2>&1

   3. Optional:If the GCC version of the test machine and the auxiliary test machine is later than 10, you must run the following command to open the nettest_omni.c file and manually delete the declared variables:

      cd netperf
      vim src/nettest_omni.c

      Manually delete the following declared variables in the nettest_omni.c file.

      /* different options for the sockets        */
      
      int
        loc_nodelay,    /* don't/do use NODELAY  locally    */
        rem_nodelay,    /* don't/do use NODELAY remotely  */
        loc_sndavoid,    /* avoid send copies locally    */
        loc_rcvavoid,    /* avoid recv copies locally    */
        rem_sndavoid,    /* avoid send copies remotely    */
        rem_rcvavoid;   /* avoid recv_copies remotely    */

      After you delete the previous variables, press the Esc key. Then, enter :wq and press the Enter key to save the modification and exit.
   4. Run the following command to install Netperf and SAR:

      cd netperf
      ./configure
      make && make install

4. Run the following command on the test machine to start 64 netserver services:

   #!/bin/bash
   for j in `seq 64`; do
       netserver -p $[16000+j] > server_$[16000+j].netperf > /dev/null 2>&1  &
   done

5. Run the following command on the test machine to query the private IP address of the test machine:

   ifconfig || ip addr

   
6. Run the following command on the auxiliary test machine to send traffic to the test machine:

   #!/bin/bash
   server_ip=<Private IP address of the test machine>
   for j in `seq 64`; do
       port=$[16000+j]
       netperf -H ${server_ip} -l ${run_time:-300} -t TCP_STREAM -p $port -- -D > /dev/null 2>&1 & 
   done

   Replace <Private IP address of the test machine> with the private IP address of the test machine obtained in the previous step, as shown in the following figure.
7. Run the following command on the test machine to test the network traffic:

   sar -n DEV 1

   View the values in the rxpck/s column in the test result. The rxpck/s column shows the total number of packets received by the test machine per second. In this example, the number of packets received by the test machine per second is about 900,000, as shown in the following figure.

Test the packet forwarding rate (greater than 6 million pps and less than 20 million pps)

1. Connect to the test machine and the auxiliary test machine.
   For more information about the methods for connecting to an instance, see Connection methods.
2. Run the following command on three auxiliary test machines and one test machine to install sockperf:

   yum install -y autoconf automake libtool gcc-c++
   cd /opt
   wget https://github.com/Mellanox/sockperf/archive/refs/tags/3.8.tar.gz
   tar -zxf 3.8.tar.gz
   cd sockperf-3.8/
   ./autogen.sh
   ./configure
   make -j `cat /proc/cpuinfo| grep process | wc -l`
   make install

3. Run the following command on three auxiliary test machines to send traffic to the test machine:

   server_ip=<Private IP address of the test machine>
   threads=64
   msg_size=14
   run_time=60
   
   basePort=6666
   for((i=0;i<$threads;++i));do
       nohup sockperf tp -i $server_ip --pps max -m ${msg_size} -t ${run_time} --port $[${basePort}+${i}] 2>&1 &
   done

   Replace <Private IP address of the test machine> with the private IP address of the actual test machine. run_time specifies the time period during which traffic is sent.
4. Run the following command on three auxiliary test machines and one test machine to test the network traffic:

   sar -n DEV 1

   View the values in the rxpck/s column on the test machine. The rxpck/s column shows the total number of packets received by the test machine per second. In this example, the number of packets received by the test machine per second is about 12,000,000, as shown in the following figure.

Test the packet forwarding rate (greater than 20 million pps)

1. Connect to the test machine and the auxiliary test machine.
   For more information about the methods for connecting to an instance, see Connection methods.
2. Run the following command on three auxiliary test machines and one test machine to install sockperf:

   yum install -y autoconf automake libtool gcc-c++
   cd /opt
   wget https://github.com/Mellanox/sockperf/archive/refs/tags/3.8.tar.gz
   tar -zxf 3.8.tar.gz
   cd sockperf-3.8/
   ./autogen.sh
   ./configure
   make -j `cat /proc/cpuinfo| grep process | wc -l`
   make install

3. Run the following command on the test machine to bind the interrupts:

   a=$(cat /proc/interrupts | grep virtio2-input | awk -F ':' '{print $1}')
   cpu=0
   for irq in $a; do
       echo $cpu >/proc/irq/$irq/smp_affinity_list
       let cpu+=2
   done

4. Run the following command on three auxiliary test machines to send traffic to the test machine:

   server_ip=<Private IP address of the test machine>
   threads=64
   msg_size=14
   run_time=60
   
   basePort=6666
   for((i=0;i<$threads;++i));do
       nohup sockperf tp -i $server_ip --pps max -m ${msg_size} -t ${run_time} --port $[${basePort}+${i}] 2>&1 &
   done

   Replace <Private IP address of the test machine> with the private IP address of the actual test machine. run_time specifies the time period during which traffic is sent.
5. Run the following command on three auxiliary test machines and one test machine to test the network traffic:

   sar -n DEV 1

   View the values in the rxpck/s column on the test machine. The rxpck/s column shows the total number of packets received by the test machine per second. In this example, the number of packets received by the test machine per second is about 20,000,000, as shown in the following figure.

Test the network bandwidth

1. Connect to the test machine and the auxiliary test machine.
   For more information about the methods for connecting to an instance, see Connection methods.
2. Run the following command on the test machine and the auxiliary test machine to download Netperf:

   wget https://benchmark-packages.oss-cn-qingdao.aliyuncs.com/netperf-2.7.0.tar.gz

3. Run the following command on the test machine and the auxiliary test machine to install Netperf and SAR:

   yum install -y gcc autoconf automake libtool sysstat
   tar -zxvf netperf-2.7.0.tar.gz
   cd netperf
   ./configure
   make && make install

4. Run the following command on the test machine to start 64 netserver services:

   #!/bin/bash
   for j in `seq 64`; do
       netserver -p $[16000+j] > server_$[16000+j].netperf > /dev/null 2>&1  &
   done

5. Run the following command on the test machine to query the private IP address of the test machine:

   ifconfig || ip addr

   
6. Run the following command on the auxiliary test machine to send traffic to the test machine:

   #!/bin/bash
   server_ip=<Private IP address of the test machine>
   for j in `seq 64`; do
       port=$[16000+j]
       netperf -H ${server_ip} -l ${run_time:-300} -t TCP_STREAM -p $port -- -D > /dev/null 2>&1 & 
   done

   Replace <Private IP address of the test machine> with the private IP address of the test machine obtained in the previous step, as shown in the following figure.
7. Run the following command on the test machine to test the network bandwidth:

   sar -n DEV 1

   View the values in the rxkB/s column in the test result. The rxkB/s column shows the total number of bytes that are received by the test machine per second. The network bandwidth (unit: Kbit/s) is the value of rxkB/s multiplied by 8.

Test the network latency

1. Connect to the test machine and the auxiliary test machine.
   For more information about the methods for connecting to an instance, see Connection methods.
2. Run the following command on the test machine and the auxiliary test machine to install sockperf:

   yum install -y autoconf automake libtool gcc-c++
   cd /opt
   wget https://github.com/Mellanox/sockperf/archive/refs/tags/3.8.tar.gz
   tar -zxf 3.8.tar.gz
   cd sockperf-3.8/
   ./autogen.sh
   ./configure
   make -j `cat /proc/cpuinfo| grep process | wc -l`
   make install

3. Run the following command on the test machine to start sockperf:

   sockperf sr --tcp --daemonize

4. Run the following command on the test machine to query the private IP address of the test machine:

   ifconfig || ip addr

   
5. Run the following command on the auxiliary test machine to send traffic to the test machine:

   sockperf pp -i <Private IP address of the test machine> --tcp -t 30 -m 14 --full-log=result.json 

   Replace <Private IP address of the test machine> with the private IP address of the test machine obtained in the previous step.

   View the data returned, as shown in the following figure.

   • Data returned staring with avg-latency indicates the average latency in microseconds.
   • Data returned staring with percentile 99.000 indicates the 99-percentile latency in microseconds.