The network interface controller (NIC) multi-queue feature allows you to configure multiple transmit (Tx) and receive (Rx) queues on a NIC. Each queue can be processed by a different CPU core. The NIC multi-queue feature is designed to improve network I/O throughput and reduce latency by allowing multiple CPU cores to simultaneously process network packets in different queues on a NIC.
Benefits of the NIC multi-queue feature
A traditional single-queue NIC uses a single CPU core to process all packets, which easily leads to CPU overload, increased latency, and packet loss. Modern servers are equipped with multi-core CPUs. A multi-queue NIC can distribute network traffic to different CPU cores. This improves the utilization of the multi-core computing resources.
Test results indicate that under identical packet forwarding rate and network bandwidth conditions, two queues outperform a single queue by 50% to 100% and performance improvements are significantly higher when four queues are used. The NIC multi-queue feature provides the following benefits:
Improve the utilization of the multi-core CPU architecture. A multi-queue NIC can distribute network traffic to multiple CPU cores for processing. This can better load-balance CPU cores and improve CPU utilization.
Improve throughput. A multi-queue NIC allows multiple packets to be simultaneously processed, which can significantly improve the throughput of the network, especially under high loads.
Reduce latency. Packets can be distributed to different queues for processing. This reduces congestion in a single queue and reduces latency.
Reduce packet loss. In high-traffic scenarios, the NIC multi-queue feature can reduce packet loss caused by the overload of a single queue.
In most cases, the NIC multi-queue feature provides the preceding benefits. However, improper configuration of the feature may cause performance degradation or other issues. For example, improper configuration of the number of queues supported by a NIC and CPU affinity may cause unnecessary context switching overheads. If you set the number of queues supported by a NIC to a small value, you cannot make full use of hardware resources.
In most cases, after you bind an elastic network interface (ENI) to an Elastic Compute Service (ECS) instance, the number of queues supported by the ENI is automatically set to the default number of queues supported by the instance type per ENI and takes effect in the operating system. If you want to manually change the number of queues supported by an ENI, consider how to properly configure the multi-queue parameters of the ENI based on your business scenario and hardware conditions.
Core mechanisms of the NIC multi-queue feature
Queue division
An ENI supports multiple Combined queues. Each Combined queue is processed by an independent CPU core. This way, packets in different queues can be processed in parallel. This reduces lock contention and fully utilizes multi-core performance.
Rx queues and Tx queues are two different types of queues used by ENIs to process packets. Each Combined queue consists of an Rx queue and a Tx queue.
Rx queue: is used to process packets received from the network. When packets arrive at an ENI, the packets are distributed to different Rx queues based on specific rules, such as polling and flow-based distribution.
Tx queue: manages the packets to be sent. Output requests generated by applications are first placed into the corresponding Tx queues and then sent by an ENI based on various factors such as order or priority.
Support for IRQ Affinity
Each queue is associated with an independent interrupt. Interrupt Request (IRQ) Affinity is used to distribute interrupts for different queues to specific CPU cores to prevent the overload of a single CPU core.
By default, IRQ Affinity is enabled in all images without the need for additional configurations, except Red Hat Enterprise Linux images. For more information, see the Configure IRQ Affinity section of this topic.
Constraints and limits
For information about the instance types that support the NIC multi-queue feature, see Overview of instance families. If the value in the NIC queues column for an instance type is greater than 1, the instance type supports the NIC multi-queue feature.
For an instance type that supports the NIC multi-queue feature, the feature automatically takes effect on an instance of the instance type after you bind an ENI to the instance.
The value in the NIC queues column for an instance type in the instance type tables indicates the maximum number of queues allowed per ENI for the instance type.
You can call the DescribeInstanceTypes operation to query the metrics about the number of NIC queues for the instance family specified by the InstanceTypeFamily parameter.
The default number of queues supported by an ENI
The value of the PrimaryEniQueueNumber parameter in the response indicates the default number of queues supported by the primary ENI. The value of the SecondaryEniQueueNumber parameter indicates the default number of queues supported by a secondary ENI.
The maximum number of queues supported by an ENI
The value of the MaximumQueueNumberPerEni parameter in the response indicates the maximum number of queues allowed per ENI for the specified instance family.
The queue quota
The value of the TotalEniQueueQuantity parameter in the response indicates the queue quota of the specified instance family.
Specific early-version public images that contain kernel versions earlier than 2.6 may not support the NIC multi-queue feature. We recommend that you use the latest public images.
View the number of queues supported by an ENI
View the number of queues supported by an ENI in the ECS console
In the top navigation bar, select the region and resource group of the resource that you want to manage.
Click the ID of the secondary ENI that you want to manage to go to the ENI details page.
In the Basic Information section, view the value of the Queues parameter, which indicates the number of queues supported by the ENI.
If you changed the number of queues supported by the ENI, the new number of queues is displayed.
If you never change the number of queues supported by the ENI, one of the following values is displayed:
An empty value is displayed if the ENI is not bound to an ECS instance.
The default number of queues supported by the ENI is displayed if the ENI is bound to an ECS instance.
View the number of queues supported by an ENI by calling an API operation
Call the DescribeNetworkInterfaceAttribute operation to query the queue-related parameters of an ENI and check the value of the QueueNumber parameter in the response to obtain the number of queues supported by the ENI.
If you changed the number of queues supported by the ENI, the new number of queues is displayed.
If you never change the number of queues supported by the ENI, one of the following values is displayed:
An empty value is displayed if the ENI is not bound to an ECS instance.
The default number of queues supported by the ENI is displayed if the ENI is bound to an ECS instance.
View the number of queues supported by an ENI after you connect to the associated ECS instance
Connect to a Linux ECS instance.
NoteYou can view the number of queues supported by an ENI that is bound to a Windows ECS instance in the ECS console or by calling an API operation.
For more information, see Use Workbench to connect to a Linux instance over SSH.
Run the
ip a
command to view the network configurations.Run the following command to check whether the NIC multi-queue feature is enabled on the primary ENI eth0.
In this example, the primary ENI is used. If you use a secondary ENI, replace the ENI name with the actual ENI name, such as eth1 or eth2.
ethtool -l eth0
View the command output to check whether the NIC multi-queue feature is enabled.
If the value of the Combined field in the Pre-set maximums section is greater than 1, the ENI supports the multi-queue feature. The value of the Combined field in the Pre-set maximums section indicates the maximum number of queues supported by the ENI.
The value of the Combined field in the Current hardware settings section indicates the number of queues that can be used by the ENI.
In this example, the command output indicates that the ENI supports up to three Combined queues and uses three Combined queues.
For information about how to change the maximum number of queues supported by an ENI, see the Change the number of queues supported by an ENI section of this topic.
For information about how to change the number of queues that can be used by an ENI, see the Change the number of queues that can be used by an ENI in the operating system section of this topic.
Change the number of queues supported by an ENI
In most cases, after you bind an ENI to an ECS instance, the number of queues supported by the ENI is automatically set to the default number of queues supported by an ENI for the instance type and takes effect in the operating system. If you want to manually change the number of queues supported by an ENI, you can use the ECS console or call an API operation. Take note of the following items:
You can change the number of queues supported by an ENI only if the ENI is in the Available state or if the ENI is in the InUse state and the ECS instance to which the ENI is bound is in the Stopped state.
The number of queues supported by an ENI that is bound to an ECS instance cannot exceed the maximum number of queues allowed per ENI for the instance type.
The total number of queues for all ENIs on an ECS instance cannot exceed the queue quota allowed for the instance type.
Use the ECS console
In the top navigation bar, select the region and resource group of the resource that you want to manage.
Click the ID of the secondary ENI that you want to manage to go to the ENI details page.
Change the number of queues supported by the ENI.
Click Confirm.
Call an API operation
Call the ModifyNetworkInterfaceAttribute operation and specify a value for the QueueNumber parameter to change the number of queues supported by an ENI.
If you change the number of queues supported by an ENI that is bound to an ECS instance, the new number of queues takes effect after the instance starts.
Change the number of queues that can be used by an ENI in the operating system
After you change the number of queues supported by an ENI in the ECS console or by calling an API operation, the change automatically takes effect in the operating system.
You can change the number of queues that can be used by an ENI in the operating system to a number that is smaller than the number of queues supported by the ENI.
When you change the number of queues that can be used by an ENI in the operating system, the change does not affect the number of queues supported by the ENI. The number of queues that can be used by an ENI is not displayed in the ECS console or returned in the response of an API operation.
If you change the number of queues that can be used by an ENI in the operating system, the change does not take effect after the instance restarts.
In the following example, an ECS instance that runs Alibaba Cloud Linux 3 is used. Alibaba Cloud Linux 3 supports the NIC multi-queue feature.
Connect to a Linux ECS instance.
For more information, see Use Workbench to connect to a Linux instance over SSH.
Run the
ip address show
command to view the network configurations.Run the following command to check whether the NIC multi-queue feature is enabled on the primary ENI eth0.
In this example, the primary ENI is used. If you use a secondary ENI, replace the ENI name with the actual ENI name, such as eth1 or eth2.
ethtool -l eth0
View the command output to check whether the NIC multi-queue feature is enabled.
If the value of the Combined field in the Pre-set maximums section is greater than 1, the ENI supports the multi-queue feature. The value of the Combined field in the Pre-set maximums section indicates the maximum number of queues supported by the ENI.
The value of the Combined field in the Current hardware settings section indicates the number of queues that can be used by the ENI.
In this example, the command output indicates that the ENI supports up to three Combined queues and uses three Combined queues.
Run the following command to change the number of queues that can be used by the primary ENI eth0 to 2.
In this example, the primary ENI is used. If you use a secondary ENI, replace the ENI name with the actual ENI name, such as eth1 or eth2.
sudo ethtool -L eth0 combined N
N
specifies the number of queues that you want the ENI to use.N
must be less than or equal to the value of the Combined field in the Pre-set maximums section.In this example, the primary ENI is configured to use two queues.
sudo ethtool -L eth0 combined 2
Configure IRQ Affinity
When you use the NIC multi-queue feature, you must configure IRQ Affinity to assign interrupts for different queues to specific CPUs, instead of allowing the interrupts to be assigned to arbitrary CPUs. This helps reduce contention among CPUs and improve network performance.
By default, IRQ Affinity is enabled in all images without the need for additional configurations, except Red Hat Enterprise Linux images.
IRQ Affinity is supported by Red Hat Enterprise Linux images but is disabled by default for the images. To configure IRQ Affinity for instances that use Red Hat Enterprise Linux images, perform the operations that are described in this section.
This section describes how to use the ecs_mq
script to configure IRQ Affinity for a Linux ECS instance that uses a Red Hat 9.2 image. If the image is not a Red Hat Enterprise Linux image, IRQ Affinity is enabled by default. You do not need to configure IRQ Affinity.
Connect to a Linux ECS instance.
For more information, see Use Workbench to connect to a Linux instance over SSH.
(Optional) Disable the irqbalance service.
The irqbalance service dynamically modifies IRQ Affinity configurations. The
ecs_mq
script conflicts with the irqbalance service. We recommend that you disable the irqbalance service.systemctl stop irqbalance.service
Run the following command to download the package that contains the new version of the
ecs_mq
script:wget https://ecs-image-tools.oss-cn-hangzhou.aliyuncs.com/ecs_mq/ecs_mq_2.0.4.tgz
Run the following command to extract the
ecs_mq
script:tar -xzf ecs_mq_2.0.tgz
Run the following command to change the working path:
cd ecs_mq/
Run the following command to install the environment that is required to run the
ecs_mq
script:bash install.sh redhat 9
NoteReplace redhat and 9 with the actual
operating system name
andmajor version number of the operating system
.Run the following command to start the
ecs_mq
script:systemctl start ecs_mq
After the script is started, IRQ Affinity is automatically enabled.
You can change the number of queues on ENIs, configure IRQ Affinity, or perform both preceding operations to optimize network performance. To ensure load balancing, assign an appropriate number of queues to each CPU core and configure IRQ Affinity based on the actual loads and performance data of the system, such as throughput and latency. To obtain the system performance data, you can test various queue and IRQ Affinity configurations.