Configure QoS for pods - Container Service for Kubernetes

Online services are sensitive to network quality and require low latency. In contrast, offline services consume a large amount of network bandwidth. If these services run together without any controls, they can compete for network resources. This competition may lead to high service latency or service unavailability. You can customize the inbound and outbound bandwidth and data packet priority for pods to achieve more granular control over network resources. This helps reduce the impact of services competing for bandwidth.

Pod bandwidth limits

You can use standard Kubernetes pod annotations to limit the inbound and outbound bandwidth for a pod:

Annotation	Note
`kubernetes.io/ingress-bandwidth: 10M`	The inbound bandwidth limit for the pod. An example value is 10 Mbps.
`kubernetes.io/egress-bandwidth: 10M`	The outbound bandwidth limit for the pod. An example value is 10 Mbps.

Important

The kubernetes.io/ingress-bandwidth annotation is not supported when Terway is configured in shared Elastic Network Interface (ENI) mode and the IPvlan+eBPF network acceleration mode is enabled.
The kubernetes.io/ingress-bandwidth annotation is not supported when Terway is configured in exclusive ENI mode.
In DataPathV2 network acceleration mode, Terway v1.13.0 and later support Ingress bandwidth limits. These bandwidth limits apply only to traffic that enters or leaves a node. The limits do not apply to traffic within the same node.
For clusters created with a Terway version earlier than v1.3.0, modify the Terway configuration to enable this feature. For more information, see the instructions below.

For clusters created before Terway v1.3.0

If your cluster was created with a Terway version earlier than v1.3.0, modify the Terway configuration to enable these annotations.

Modify the Terway configuration

Run the following command to edit the Terway ConfigMap.
```
kubectl edit cm -n kube-system eni-config
```

In data.10-terway.conf, add the capabilities configuration.

apiVersion: v1
data:
  10-terway.conf: |
    {
      "cniVersion": "0.3.1",
      "name": "terway",
      "capabilities": {"bandwidth": true},
      "type": "terway"
    }

Save the configuration and exit. Then, run the following command to restart the Terway pods and apply the changes.
```
kubectl delete -n kube-system pod -l app=terway-eniip
```

Packet priority control policy

By default, the network interface card (NIC) queue sends packets on a first-in, first-out (FIFO) basis. Terway lets you configure a packet priority control policy for pods. This ensures that packets for high-priority services are sent and received first during peak traffic, which reduces latency.

This feature is disabled by default. You can enable it by modifying the Terway configuration.

Important

This feature is not supported for clusters that have the DataPathV2 network acceleration mode enabled.
This feature is not supported for nodes in exclusive ENI mode.
After you enable priority control, the ENI's NIC queue is replaced with mq+prio.

Enable packet priority control

Run the following command to edit the Terway ConfigMap.
```
kubectl edit cm -n kube-system eni-config
```

In data.10-terway.conf, add the enable_network_priority configuration.

apiVersion: v1
data:
  10-terway.conf: |
    {
      "cniVersion": "0.3.1",
      "name": "terway",
      "enable_network_priority": true,
      "type": "terway"
    }

Save the configuration and exit. Then, run the following command to restart the Terway pods and apply the changes.
```
kubectl delete -n kube-system pod -l app=terway-eniip
```

Use the following pod annotations to specify the priority of pod packets.

Description	Note
k8s.aliyun.com/network-priority: "guaranteed"	Highest priority, for latency-sensitive services
k8s.aliyun.com/network-priority: "best-effort"	Medium priority, suitable for general services.
k8s.aliyun.com/network-priority: "burstable"	Lowest priority, for services that require high bandwidth but are not sensitive to latency.