This topic describes how to replace a Userspace I/O (UIO) driver with a Virtual Function I/O (VFIO) driver to solve the runtime exceptions of Data Plane Development Kit (DPDK) for an Elastic Compute Service (ECS) instance.

Prerequisites

  • Huge pages are configured for the instance.
  • DPDK is installed on the instance.
  • You are connected to the instance. For more information, see Guidelines on instance connection.

Background information

If DPDK applications are deployed on your ECS instances of a sixth-generation (such as g6, c6, and r6) instance type or later, exceptions may occur when you run the applications. For example, the igb_uio port to which the network interface controllers (NICs) are bound may not be detected when you use Pktgen-DPDK to test the packet forwarding rate of an instance, and the following error message is reported:
EAL: eal_parse_sysfs_value(): cannot open sysfs value /sys/bus/pci/devices/0000:00:06.0/uio/uio0/portio/port0/start

You can replace UIO drivers with VFIO drivers to solve these problems. Operations performed on ECS bare metal instances are different from those performed on instances that are not ECS bare metal instances. For more information, see Operations performed on an instance that is not an ECS bare metal instance and Operations performed on an ECS bare metal instance.

Operations performed on an instance that is not an ECS bare metal instance

  1. Run the following command to check the configurations of GRand Unified Bootloader (GRUB):
    cat /proc/cmdline

    Check whether the configurations of GRUB contain intel_iommu=on and do not contain iommu=pt.

  2. Optional:If the configurations of GRUB do not contain intel_iommu=on, manually add intel_iommu=on.
    1. Run the following command to open the configuration file of GRUB:
      vim /etc/default/grub
    2. Switch to the edit mode, add intel_iommu=on to the line of GRUB_CMDLINE_LINUX, and then save the configuration file.
      The following figure shows an example of the modified configuration file. grub-config
    3. Run the following command to apply the modified configurations:
      grub2-mkconfig -o /boot/grub2/grub.cfg
    4. Restart the instance and connect to the instance.
  3. Run the following commands to install the VFIO and VFIO-PCI drivers:
    modprobe vfio && \
    modprobe vfio-pci
  4. Run the following command to configure noiommu_mode:
    echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
  5. Run the following command to view and record the bus-info value of the NIC bound to the instance:
    ethtool -i ethX
    Replace ethX with the ID of the NIC bound to the instance. In this example, eth1 is used. bus-info
  6. Switch to the usertools directory under the installation directory of DPDK and run the following command to bind the NIC to the VFIO-PCI driver:
    ./dpdk-devbind.py -b vfio-pci 0000:00:06.0
    Note In this example, 0000:00:06.0 is the bus-info value of eth1. In actual scenarios, replace it with the bus-info value of your NIC.
    Run the ./dpdk-devbind.py --status command to view the binding state of the NIC. The following command output indicates that the NIC whose bus-info value is 0000:00:06.0 has been bound to the VFIO-PIC driver. bound-status
  7. Switch to the build/app directory under the installation directory of DPDK and run one of the following commands to start DPDK.
    • If the version of DPDK is 18.02 or later, run the following command:
      ./testpmd -w 0000:00:06.0 -c 0x3 -- --total-num-mbufs=2048 -ai
    • If the version of DPDK is earlier than 18.02, run the following command:
      ./testpmd -w 0000:00:06.0 -c 0x3 -- --total-num-mbufs=2048 --disable-hw-vlan -ai
    Note -w specifies the bus-info value of an NIC. In this example, -w is set to 0000:00:06.0. total-num-mbufs specifies the huge page size. In this example, total-num-mbufs is set to 2048. In actual scenarios, replace them based on your needs.
    The following figure shows the sample command output after DPDK is run. run-dpdk

Operations performed on an ECS bare metal instance

  1. Run the following command to check the configurations of GRUB:
    cat /proc/cmdline

    Check whether the configurations of GRUB contain intel_iommu=on and iommu=pt.

  2. Optional:If the configurations of GRUB do not contain intel_iommu=on and iommu=pt, manually add intel_iommu=on and iommu=pt.
    1. Run the following command to open the configuration file of GRUB:
      vim /etc/default/grub
    2. Switch to the edit mode, add intel_iommu=on and iommu=pt to the line of GRUB_CMDLINE_LINUX, and then save the configuration file.
      The following figure shows an example of the modified configuration file. ebm-grub
    3. Run the following command to apply the modified configurations:
      grub2-mkconfig -o /boot/grub2/grub.cfg
    4. Restart the instance and connect to the instance.
  3. Run the following commands to install the VFIO and VFIO-PCI drivers:
    modprobe vfio && \
    modprobe vfio-pci
  4. Run the following command to configure noiommu_mode:
    echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
  5. Run the following command to view and record the bus-info value of the NIC bound to the instance:
    ethtool -i ethX
    Replace ethX with the ID of the NIC bound to the instance. In this example, eth1 is used. ebm-bus-info
  6. Switch to the usertools directory under the installation directory of DPDK and run the following command to bind the NIC to the VFIO-PCI driver:
    ./dpdk-devbind.py -b vfio-pci 0000:b7:00.0
    Note In this example, 0000:b7:00.0 is the bus-info value of eth1. In actual scenarios, replace it with the bus-info value of your NIC.

    Run the ./dpdk-devbind.py --status command to view the binding state of the NIC. The following command output indicates that the NIC whose bus-info value is 0000:b7:00.0 has been bound to the VFIO-PIC driver.

    bound-eth1
  7. Switch to the build/app directory under the installation directory of DPDK and run one of the following commands to start DPDK.
    • If the version of DPDK is 18.02 or later, run the following command:
      ./testpmd -w 0000:b7:00.0 -c 0x3 -- --total-num-mbufs=2048 -ai
    • If the version of DPDK is earlier than 18.02, run the following command:
      ./testpmd -w 0000:b7:00.0 -c 0x3 -- --total-num-mbufs=2048 --disable-hw-vlan -ai
    Note -w specifies the bus-info value of an NIC. In this example, -w is set to 0000:b7:00.0. total-num-mbufs specifies the huge page size. In this example, total-num-mbufs is set to 2048. In actual scenarios, replace them based on your needs.

    The following figure shows the sample command output after DPDK is run.

    ebm-testpmd