:Test ECS network performance with Pktgen

Step 1: Prepare the environment

Prepare two servers: a sender and a receiver. Add a network interface controller to each server—one for logon and one for testing. Ensure that the two test network interface controllers can communicate with each other. Then, run the ifconfig command to record the IP address and MAC address of both test network interface controllers. You will need this information to configure the traffic stream.

Step 2: Install the software

CentOS 6 reached end of life (EOL). In accordance with Linux community rules, all content was removed from the following CentOS 6 repository address: http://mirror.centos.org/centos-6/. If you continue to use the default CentOS 6 repository on Alibaba Cloud, an error is reported. To use specific installation packages of CentOS 6, change the CentOS 6 repository address. For more information, see How do I change CentOS 6 repository addresses?

Perform the following steps on both the sender and the receiver.

1. Connect to the Linux ECS instance. For details, see Connect to a Linux instance using Workbench.

2. Run the following commands to create the pktgen directory.

   cd /root/
   mkdir pktgen

3. Run the following command to change to the pktgen directory.

   cd pktgen

4. Run the following command to download the Pktgen, Lua, and DPDK installation packages.

   wget https://git.dpdk.org/apps/pktgen-dpdk/snapshot/pktgen-dpdk-pktgen-3.6.5.zip https://www.lua.org/ftp/lua-5.4.3.tar.gz https://fast.dpdk.org/rel/dpdk-18.11.11.tar.xz

   Note

   Note: The versions of Pktgen, Lua, and DPDK must be compatible to avoid compilation errors. This topic uses the versions listed as an example, but you can use other compatible versions.

5. Run the following commands to decompress the installation packages.

   tar -Jxvf dpdk-18.11.11.tar.xz
   tar -zxvf lua-5.4.3.tar.gz
   unzip pktgen-dpdk-pktgen-3.6.5.zip

6. If dependencies are missing during compilation, run the commands for your operating system to install the required packages.

   • CentOS 7

     yum -y install kernel-devel.x86_64
      yum -y install numactl-devel.x86_64
      yum -y install elfutils-libelf-devel
      yum -y install libpcap-devel

   • CentOS 8

     Note

     In CentOS 8, the libpcap-devel package is not installed by default. You must install it manually.

     Before running the following commands, switch the mirror repository as instructed in How to switch the mirror repository for CentOS 8 after EOL?. Then, add the following content to the CentOS-PowerTools.repo file that you created.

     [PowerTools]
     name=CentOS-8.5.2111 - PowerTools - mirrors.aliyun.com
     baseurl=http://mirrors.cloud.aliyuncs.com/centos-vault/8.5.2111/PowerTools/$basearch/os/
             http://mirrors.aliyuncs.com/centos-vault/8.5.2111/PowerTools/$basearch/os/
             http://mirrors.aliyun.com/centos-vault/8.5.2111/PowerTools/$basearch/os/
     gpgcheck=0
     enabled=1
     gpgkey=http://mirrors.cloud.aliyuncs.com/centos/RPM-GPG-KEY-CentOS-Official

     dnf config-manager --add-repo /etc/yum.repos.d/CentOS-PowerTools.repo 
     dnf config-manager --set-enabled PowerTools
     dnf -y upgrade libmodulemd
     dnf -y install kernel-devel.x86_64
     dnf -y install elfutils-libelf-devel
     dnf -y install numactl-devel.x86_64
     dnf -y install libpcap-devel

7. Run the following commands to compile and install DPDK.

   cd dpdk-stable-18.11.11
   make config T=x86_64-native-linuxapp-gcc 
   make all -j32
   cd ..

8. Run the following commands to compile and install Lua.

   cd lua-5.4.3
   make
   make install
   cd ..

9. Run the following commands to compile and install Pktgen.

   cd pktgen-dpdk-pktgen-3.6.5/
    export RTE_SDK=/root/pktgen/dpdk-stable-18.11.11/
    export RTE_TARGET=build 
   make -j32

Step 3: Bind the driver to VFIO

Perform the following steps on both the sender and the receiver.

1. Run the following command to check the /proc/cmdline file and ensure that the GRUB configuration for the operating system includes intel_iommu=on.

   cat /proc/cmdline

   The system displays output similar to the following. If the parameter is not present, follow these steps to add it.

   BOOT_IMAGE=/boot/vmlinuz-3.10.0-XXXX.19.1.el7.x86_64 root=UUID=9f2d3e15-XXXX-XXXX-8385-0165b4b67864 ro crashkernel=auto spectre_v2=retpoline rhgb quiet net.ifnames=0 console=tty0 console=ttyS0,115200n8 noibrs intel_iommu=on

   1. Run the following command to edit the /etc/default/grub file.

      vim /etc/default/grub

   2. Press the i key to enter insert mode. Add intel_iommu=on to the GRUB_CMDLINE_LINUX line. The modified content is shown below.

   3. To save your changes and close the file, press the Esc key, enter :wq, and press Enter.

   4. Run the following command to update the configuration.

      grub2-mkconfig -o /boot/grub2/grub.cfg

   5. Run the following command to restart the server.

      reboot

2. Run the following commands to load the vfio and vfio-pci drivers.

   modprobe vfio
   modprobe vfio-pci

3. Run the following command to configure the noniommu_mode.

   echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

4. Run the following command to view and record the bus-info value of the network interface controller.

   ethtool -i eth1

   Note

   Note: This example uses the eth1 network interface controller. Replace eth1 with the actual name in your environment.

   The system displays similar output.

   driver: virtio_net
   version: 1.0.0
   firmware-version:
   expansion-rom-version:
   bus-info: 0000:00:08.0
   supports-statistics: no
   supports-test: no
   supports-eeprom-access: no
   supports-register-dump: no
   supports-priv-flags: no

5. Run the following command to unbind the virtio driver from the eth1 network interface controller.

   Important

   This example unbinds the virtio driver from the eth1 network interface controller. This action will cause a network interruption, so we recommend performing it during off-peak hours.

   echo "[$Bus-info]" > /sys/bus/pci/drivers/virtio-pci/unbind

   Note

   Note: Replace [$Bus-info] with the bus-info value that you obtained in the previous step.

6. Run the following command to bind the bus-info to the vfio-pci driver.

   /root/pktgen/dpdk-stable-18.11.11/usertools/dpdk-devbind.py -b vfio-pci [$Bus-info]

   Note

   This example is for a CentOS 8.3 (Python 3.6.8) environment. To run the Python script, modify the command as follows: python3 /root/pktgen/dpdk-stable-18.11.11/usertools/dpdk-devbind.py -b vfio-pci [$Bus-info]

Step 4: Configure huge pages

On both the sender and the receiver, run the command that corresponds to the number of nodes on your system.

• Dual-node

  echo 8000 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
  echo 8000 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

• Single-node

  echo 8000 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

Step 5: Start Pktgen

Perform the following steps on both the sender and the receiver.