All Products
Search
Document Center

:Use Pktgen to test the network performance of ECS instances

Last Updated:Jun 24, 2021

Disclaimer: This article may contain information about third-party products. Such information is for reference only. Alibaba Cloud does not make any guarantee, express or implied, with respect to the performance and reliability of third-party products, as well as potential impacts of operations on the products.

Overview

This topic describes how to use Pktgen to test the network performance of Elastic Compute Service (ECS) instances.

Background information

Take note of the following items:

  • Before you perform risky operations such as modifying instance configurations or data, we recommend that you check the disaster recovery and fault tolerance capabilities of the instances to ensure data security.
  • You can modify the configurations and data of instances including ECS and ApsaraDB RDS instances. We recommend that before you modify instance configurations or data, you create snapshots or enable RDS log backup.
  • If you have authorized or submitted sensitive information such as the logon account and password in the Alibaba Cloud Management Console, we recommend that you modify these information in a timely manner.

The following steps describe how to use Pktgen to test the network performance of ECS instances:

Step 1: Prepare the environment

Prepare two ECS instances, one for the generator and the other for the receiver. Prepare two network interface controllers (NICs), one for logon and the other for testing. Make sure that the two NICs can communicate with each other. Then, run the ifconfig command and record the IP addresses and media access control (MAC) addresses of the two NICs, which are required when you configure traffic.

Step 2: Install software

Perform the following operations on both the generator and the receiver:

  1. Log on to a Linux instance. For more information about how to log on to the Linux instance, see Connect to a Linux instance by using a username and password.
  2. Run the following command to create the pktgen directory:
    mkdir pktgen
  3. Run the following command to go to the pktgen directory:
    cd pktgen
  4. Run the following command to download the installation packages of Pktgen, Lua, and Data Plane Development Kit (DPDK):
    wget https://git.dpdk.org/apps/pktgen-dpdk/snapshot/pktgen-dpdk-pktgen-3.6.5.zip https://www.lua.org/ftp/lua-5.4.3.tar.gz https://fast.dpdk.org/rel/dpdk-18.11.11.tar.xz
    Note: The versions of Pktgen, Lua, and DPDK must be compatible with each other. Otherwise, compilation errors may occur. In this topic, the versions in the preceding command are used. Before you use the installation packages of other versions, make sure that the versions are compatible with each other.
  5. Run the following commands in sequence to decompress the installation packages of Pktgen, Lua, and DPDK:
    tar -Jxvf dpdk-18.11.11.tar.xz
    
    tar -zxvf lua-5.4.3.tar.gz
    unzip pktgen-dpdk-pktgen-3.6.5.zip
  6. If some dependencies are missing during the compilation process, run the following commands to install the corresponding installation packages:
    • CentOS 7
      yum -y install kernel-devel.x86_64
      yum -y install numactl-devel.x86_64
      yum -y install elfutils-libelf-devel
      yum -y install libpcap-devel
    • CentOS 8
      Note: By default, libpcap-devel is not installed on the CentOS 8 operating system. Run the following commands to install libpcap-devel:
      dnf config-manager --add-repo /etc/yum.repos.d/CentOS-PowerTools.repo 
      dnf config-manager --set-enabled PowerTools
      dnf install epel-release
      dnf install libpcap-devel
  7. Run the following commands in sequence to compile and install Pktgen:
    cd dpdk-stable-18.11.11
    make config T=x86_64-native-linuxapp-gcc
    make all -j32
  8. Run the following commands in sequence to compile and install Lua:
    cd lua-5.4.3
    make
    make install
  9. Run the following commands in sequence to compile and install DPDK:
    cd pktgen-dpdk-pktgen-3.6.5/
    export RTE_SDK=/root/pktgen/dpdk-stable-18.11.11/
    export RTE_TARGET=build
    make -j32

Step 3: Bind drivers to VFIO

Perform the following operations on both the generator and the receiver:

  1. View the /proc/cmdline file and check whether the line starting with BOOT_IMAGE contains the intel_iommu=on parameter. If the line does not contain the intel_iommu=on parameter, edit the /etc/default/grub file, add the intel_iommu=on parameter to the line starting with GRUB_CMDLINE_LINUX, and then restart the ECS instances.
  2. Run the following commands in sequence to install the Virtual Function I/O (VFIO) and VFIO-PCI drivers:
    modprobe vfio
    modprobe vfio-pci
  3. Run the following command to configure noniommu_mode:
    echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
  4. Run the following command to check and record the bus-info value of the NIC:
    ethtool -i eth1
    Note: In this topic, the eth1 NIC is used. In actual operations, replace eth1 with your NIC.
    A command output similar to the following one is returned:
    driver: virtio_net

    version: 1.0.0

    firmware-version:

    expansion-rom-version:
    bus-info: 0000:00:08.0
    supports-statistics: no
    supports-test: no
    supports-eeprom-access: no
    supports-register-dump: no
    supports-priv-flags: no
  5. Run the following command to unbind the virtio driver from eth1.
    echo "[$Bus-info]" > /sys/bus/pci/drivers/virtio-pci/unbind
    Note: [$Bus-info] is the bus-info value of the NIC obtained in the previous step.
  6. Run the following command to bind bus-info to the VFIO-PCI driver:
    /root/pktgen/dpdk-stable-18.11.11/usertools/dpdk-devbind.py -b vfio-pci [$Bus-info]

Step 4: Configure Hugepages

Perform the following operations on both the generator and the receiver. Check the number of your nodes and run the corresponding commands:

  • Dual nodes
    echo 8000 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
    echo 8000 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
  • Single node
    echo 8000 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

Step 5: Start Pktgen

Perform the following operations on both the generator and the receiver:

  1. Run the following command in the codepktgen-dpdk-pktgen-3.6.5code directory to start Pktgen:
    ./app/build/pktgen -c 0x1ffff -n 4 --socket-mem 16000 -w [$Bus-info] -- -P -m "[1-16:1-16].0"
    Note:
    • 0x1ffff: the mask that indicates the CPUs used by threads. In this example, the lowest set bit of the mask is bit0, which indicates that CPU0 is used to control shell and CPU1 to CPU16 are used to generate and receive packets.
    • "[1-16:1-16].0" indicates the CPU number. For more information, see the following descriptions. The CPU number cannot exceed the CPU range within which to generate and receive packets in the mask.
      • 1-16 (before the colon): CPU number. One of 16 CPUs is used to receive packets.
      • 1-16 (after the colon): CPU number. One of 16 CPUs is used to generate packets
      • .0 indicates that the port number is 0.
    • You can modify the CPU range to improve the generation performance. Note that the more CPUs required for generating packets, the more Hugepages are needed. The Pktgen process may stop due to insufficient Hugepages.
    A command output similar to the following one indicates that Pktgen is started:
    Copyright (c) <2010-2019>, Intel Corporation. All rights reserved. Powered by DPDK
    EAL: Detected 64 lcore(s)
    EAL: Detected 1 NUMA nodes
    EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
    EAL: No free hugepages reported in hugepages-1048576kB
    EAL: Probing VFIO support...
    EAL: VFIO support initialized
    EAL: PCI device 0000:00:08.0 on NUMA socket -1
    EAL:   Invalid NUMA socket, default to 0
    EAL:   probe driver: 1af4:1000 net_virtio
    EAL:   using IOMMU type 8 (No-IOMMU)
    Lua 5.4.3  Copyright (C) 1994-2021 Lua.org, PUC-Rio
    *** Copyright (c) <2010-2019>, Intel Corporation. All rights reserved.
    *** Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<
    Initialize Port 0 -- TxQ 16, RxQ 16,  Src MAC 00:16:3e:24:9b:a4
    Port  0: Link Up - speed 10000 Mbps - full-duplex <Enable promiscuous mode>
      RX/TX processing lcore:   1 rx:  1 tx:  1
      RX/TX processing lcore:   2 rx:  1 tx:  1
      RX/TX processing lcore:   3 rx:  1 tx:  1
      RX/TX processing lcore:   4 rx:  1 tx:  1
    - Ports 0-0 of 1   <Main Page>  Copyright (c) <2010-2019>, Intel Corporation
      Flags:Port        :   P-----R--------:0
    Link State          :       <UP-10000-FD>     ----TotalRate----
    Pkts/s Max/Rx       :                 1/0                   1/0
           Max/Tx       :                 0/0                   0/0
    MBits/s Rx/Tx       :                 0/0                   0/0
    Broadcast           :                   0
    Multicast           :                   0
    Sizes 64            :                   0
          65-127        :                   0
          128-255       :                   0
          256-511       :                   0
          512-1023      :                   0
          1024-1518     :                   0
    Runts/Jumbos        :                 2/0
    ARP/ICMP Pkts       :                 2/0
    Errors Rx/Tx        :                 0/0
    Total Rx Pkts       :                   2
          Tx Pkts       :                   0
          Rx MBs        :                   0
          Tx MBs        :                   0
                        :
    Pattern Type        :             abcd...
    Tx Count/% Rate     :       Forever /100%
    Pkt Size/Tx Burst   :           64 /   64
    Port Src/Dest       :         1234 / 5678
    Pkt Type:VLAN ID    :     IPv4 / UDP:0001
    802.1p CoS/DSCP/IPP :           0/  0/  0
    VxLAN Flg/Grp/vid   :    0000/    0/    0
    IP  Destination     :         192.168.X.X
        Source          :      192.168.X.X/24
    MAC Destination     :   00:00:00:00:00:00
        Source          :   00:XX:XX:24:9b:a4
    PCI Vendor/Addr     :   1af4:1000/00:08.0
    -- Pktgen Ver: 3.6.5 (DPDK 18.11.11)  Powered by DPDK  (pid:4873) -------------
    Pktgen:/>
  2. Run the following commands in sequence in Pktgen to configure traffic for the generator:
    set 0 proto udp
    range 0 src mac start [$MAC]
    range 0 src mac min [$MAC]
    range 0 src mac inc 00:00:00:00:00:00
    range 0 dst mac start ee:ff:ff:ff:ff:ff
    range 0 dst mac min ee:ff:ff:ff:ff:ff
    range 0 dst mac inc 00:00:00:00:00:00
    range 0 src port start 10000
    range 0 src port min 10000
    range 0 src port max 10000
    range 0 src port inc 0
    range 0 dst port start 2000
    range 0 dst port min 2000
    range 0 dst port max 2255
    range 0 dst port inc 1
    range 0 proto udp
    range 0 src ip start [$Local_IP]
    range 0 src ip min [$Local_IP]
    range 0 src ip inc 0.0.0.0
    range 0 size start 64
    range 0 size min 64
    range 0 size max 64
    range 0 size inc 0
    range 0 dst ip start [$Receive_IP]
    range 0 dst ip min [$Receive_IP]
    range 0 dst ip inc 0.0.0.0
    enable all range
    Note:
    • [$MAC] is the MAC address of the generator.
    • [$Local_IP] is the IP address of the generator.
    • [$Receive_IP] is the IP address of the receiver.
  3. Run the following command in Pktgen to start the sending process in the generator:
    start 0
    View the traffic data:
    • Generator
      -iPorts 0-0 of 1   <Main Page>  Copyright (c) <2010-2019>, Intel Corporation
        Flags:Port        :   P-----R--------:0                   1/0
      Link State          :       <UP-10000-FD>     ----TotalRate----
      Pkts/s Max/Rx       :                 1/0                   1/0
             Max/Tx       :   14906304/14904704     14906304/14904704
      MBits/s Rx/Tx       :             0/10015               0/10015
      Broadcast           :                   0
      Multicast           :                   0
      Sizes 64            :                   0
            65-127        :                   0
            128-255       :                   0
            256-511       :                   0
            512-1023      :                   0
            1024-1518     :                   0
      Runts/Jumbos        :                16/0
      ARP/ICMP Pkts       :                16/0
      Errors Rx/Tx        :                 0/0
      Total Rx Pkts       :                  16
            Tx Pkts       :          1655962864
    • Receiver
      - Ports 0-0 of 1   <Main Page>  Copyright (c) <2010-2019>, Intel Corporation
        Flags:Port        :   P--------------:0
      Link State          :       <UP-10000-FD>     ----TotalRate----
      Pkts/s Max/Rx       :   14911585/14910795     14911585/14910795
             Max/Tx       :                 0/0                   0/0
      MBits/s Rx/Tx       :             10020/0               10020/0
      Broadcast           :                   0
      Multicast           :                   0
      Sizes 64            :          1862740038
            65-127        :                   0
            128-255       :                   0
            256-511       :                   0
            512-1023      :                   0
            1024-1518     :                   0
      Runts/Jumbos        :                 6/0
      ARP/ICMP Pkts       :                 6/0
      Errors Rx/Tx        :                 0/0
      Total Rx Pkts       :          2298743466
            Tx Pkts       :                   0
            Rx MBs        :             1544755
            Tx MBs        :                   0
                          :
      Pattern Type        :             abcd...
      Tx Count/% Rate     :       Forever /100%
      Pkt Size/Tx Burst   :           64 /   64
      Port Src/Dest       :         1234 / 5678
      Pkt Type:VLAN ID    :     IPv4 / TCP:0001
      802.1p CoS/DSCP/IPP :           0/  0/  0
      VxLAN Flg/Grp/vid   :    0000/    0/    0
      IP  Destination     :         192.168.X.1
          Source          :      192.168.X.X/24
      MAC Destination     :   00:00:00:00:00:00
          Source          :   00:XX:XX:24:2d:55
      PCI Vendor/Addr     :   1af4:1000/00:08.0
      -- Pktgen Ver: 3.6.5 (DPDK 18.11.11)  Powered by DPDK  (pid:4072) -------------
  4. Run the following command in the generator to stop the sending process:
    stop 0

Application scope

  • ECS