:How to use Pktgen to test the network performance of an ECS instance

Step 1: Prepare the environment

Prepare two servers: one as the sender and the other as the receiver. Add a secondary network interface card (NIC) to each server. Use one NIC for management access and the other for testing. Confirm that the two test NICs can communicate. Then, run the ifconfig command and record the IP address and MAC address of both test NICs. This information is required to configure traffic on the sender.

Step 2: Install the software

CentOS 6 reached end of life (EOL). In accordance with Linux community rules, all content was removed from the following CentOS 6 repository address: http://mirror.centos.org/centos-6/. If you continue to use the default CentOS 6 repository on Alibaba Cloud, an error is reported. To use specific installation packages of CentOS 6, change the CentOS 6 repository address. For more information, see How do I change CentOS 6 repository addresses?

Perform the following operations on both the sender and receiver instances.

1. Log on to the Linux ECS instance. For more information, see Connect to a Linux instance using Workbench.

2. Run the following command to create the pktgen directory.

   cd /root/
   mkdir pktgen

3. Run the following command to change to the pktgen directory.

   cd pktgen

4. Run the following command to download the Pktgen, Lua, and DPDK installation packages.

   wget https://git.dpdk.org/apps/pktgen-dpdk/snapshot/pktgen-dpdk-pktgen-3.6.5.zip https://www.lua.org/ftp/lua-5.4.3.tar.gz https://fast.dpdk.org/rel/dpdk-18.11.11.tar.xz

   Note

   Note: The Pktgen, Lua, and DPDK versions must be compatible to prevent compilation issues. This article uses the versions in the example command, but you can use other compatible versions.

5. Run the following commands in sequence to decompress the installation packages.

   tar -Jxvf dpdk-18.11.11.tar.xz
   
   tar -zxvf lua-5.4.3.tar.gz
   
   unzip pktgen-dpdk-pktgen-3.6.5.zip

6. If any dependencies are missing during compilation, run the following commands to install the required packages.

   • CentOS 7

     yum -y install kernel-devel.x86_64
      yum -y install numactl-devel.x86_64
      yum -y install elfutils-libelf-devel
      yum -y install libpcap-devel

   • CentOS 8

     Note

     The libpcap-devel package is not installed by default in CentOS 8 and must be installed manually.

     Before you run the following commands, change the mirror repository by following the instructions in CentOS operating system. Then, from the Centos-vault-8.5.2111.repo file, copy the [PowerTools] block and paste it into a new file named CentOS-PowerTools.repo.

     dnf config-manager --add-repo /etc/yum.repos.d/CentOS-PowerTools.repo 
     dnf config-manager --set-enabled PowerTools
     dnf -y upgrade libmodulemd
     dnf -y install kernel-devel.x86_64
     dnf -y install elfutils-libelf-devel
     dnf -y install numactl-devel.x86_64
     dnf -y install libpcap-devel

7. Run the following commands in sequence to compile and install DPDK.

   cd dpdk-stable-18.11.11
   make config T=x86_64-native-linuxapp-gcc 
   make all -j32
   cd ..

8. Run the following commands in sequence to compile and install Lua.

   cd lua-5.4.3
   make
   make install
   cd ..

9. Run the following commands in sequence to compile and install Pktgen.

   cd pktgen-dpdk-pktgen-3.6.5/
    export RTE_SDK=/root/pktgen/dpdk-stable-18.11.11/
    export RTE_TARGET=build 
   make -j32

Step 3: Bind the driver to vfio

Perform the following operations on both the sender and receiver instances.

1. Run the following command to check the /proc/cmdline file and ensure the operating system's GRUB configuration contains intel_iommu=on.

   cat /proc/cmdline

   The output should be similar to the following. If the parameter is missing, add it by following these steps:

   BOOT_IMAGE=/boot/vmlinuz-3.10.0-XXXX.19.1.el7.x86_64 root=UUID=9f2d3e15-XXXX-XXXX-8385-0165b4b67864 ro crashkernel=auto spectre_v2=retpoline rhgb quiet net.ifnames=0 console=tty0 console=ttyS0,115200n8 noibrs intel_iommu=on

   1. Run the following command to edit the /etc/default/grub file.

      vim /etc/default/grub

   2. Press i to enter edit mode and add intel_iommu=on to the GRUB_CMDLINE_LINUX line. The modified content is shown in the following figure.

   3. To save and close the file, press Esc, enter :wq, and press Enter.

   4. Run the following command to update the configuration.

      grub2-mkconfig -o /boot/grub2/grub.cfg

   5. Run the following command to restart the server.

      reboot

2. Run the following commands in sequence to load the vfio and vfio-pci drivers.

   modprobe vfio
   
   modprobe vfio-pci

3. Run the following command to configure noniommu_mode.

   echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

4. Run the following command to view and record the bus-info value of the NIC.

   ethtool -i eth1

   Note

   Note: This example uses the eth1 NIC. Use the actual NIC name for your environment.

   The system displays output similar to the following.

   driver: virtio_net
   
   version: 1.0.0
   
   firmware-version:
   
   expansion-rom-version:
   
   bus-info: 0000:00:08.0
   
   supports-statistics: no
   
   supports-test: no
   
   supports-eeprom-access: no
   
   supports-register-dump: no
   
   supports-priv-flags: no

5. Run the following command to unbind the virtio driver from the eth1 NIC.

   Important

   This example unbinds the virtio driver from the eth1 NIC. This action interrupts network connectivity on the NIC. We recommend that you perform this operation during off-peak hours.

   echo "[$Bus-info]" > /sys/bus/pci/drivers/virtio-pci/unbind

   Note

   Note: Replace [$Bus-info] with the bus-info value you recorded in step 4.

6. Run the following command to bind the bus-info to the vfio-pci driver.

   /root/pktgen/dpdk-stable-18.11.11/usertools/dpdk-devbind.py -b vfio-pci [$Bus-info]

   Note

   Replace [$Bus-info] with the bus-info value you recorded in step 4. In a CentOS 8.3 (Python 3.6.8) environment, you must use the python3 interpreter to run the script. Modify the command as follows: python3 /root/pktgen/dpdk-stable-18.11.11/usertools/dpdk-devbind.py -b vfio-pci [$Bus-info]

Step 4: Configure huge pages

Perform the following operations on both the Sender and Receiver instances. Check the number of NUMA nodes on your system and run the corresponding command.

• For a dual-node system

  echo 8000 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
  echo 8000 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

• For a single-node system

  echo 8000 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

Step 5: Start Pktgen

Perform the following operations on both the sender and receiver instances.