Configure QoS for Pods - Container Service for Kubernetes

Online workloads are sensitive to network quality and require low latency. Offline workloads often consume large amounts of network bandwidth. If you run both types of workloads together without any controls, resource contention can cause high latency or even service outages. You can customize a pod's ingress and egress bandwidth and packet priority for fine-grained control over network resources and to mitigate bandwidth contention between services.

Pod bandwidth limits

You can use standard Kubernetes pod annotations to limit the ingress and egress bandwidth of a pod:

Annotation	Description
`kubernetes.io/ingress-bandwidth: 10M`	The ingress bandwidth limit for the pod. The example value is 10 Mbps.
`kubernetes.io/egress-bandwidth: 10M`	The egress bandwidth limit for the pod. The example value is 10 Mbps.

Important

When Terway is configured in shared ENI mode with the IPvlan+eBPF network acceleration mode enabled, the kubernetes.io/ingress-bandwidth pod annotation is not supported.
When Terway is configured in exclusive ENI mode, the kubernetes.io/ingress-bandwidth pod annotation is not supported.
In DataPath V2 network acceleration mode, Terway supports ingress bandwidth limits starting from v1.13.0. These limits apply only to traffic between nodes (inter-node), not traffic within a single node (intra-node).
If your cluster was created with a Terway version earlier than v1.3.0, you must modify the Terway configuration to enable this feature.

Clusters created before Terway v1.3.0

To use these annotations on a cluster created with a Terway version earlier than v1.3.0, you must modify the Terway configuration.

Modify the Terway configuration

Run the following command to edit the Terway ConfigMap.
```
kubectl edit cm -n kube-system eni-config
```

In data.10-terway.conf, add the capabilities configuration.

apiVersion: v1
data:
  10-terway.conf: |
    {
      "cniVersion": "0.3.1",
      "name": "terway",
      "capabilities": {"bandwidth": true},
      "type": "terway"
    }

Save the file, then run the following command to restart the Terway pods and apply the changes.
```
kubectl delete -n kube-system pod -l app=terway-eniip
```

Traffic prioritization policy

By default, the ENI queue processes packets based on a first-in, first-out (FIFO) rule. Terway allows you to configure a traffic prioritization policy for pod packets. This ensures that packets from high-priority services are handled first during traffic peaks, reducing latency.

This feature is disabled by default. Modify the Terway configuration to enable it.

Important

This feature is not supported for clusters with the DataPath V2 network acceleration mode enabled.
This feature is not supported for nodes in exclusive ENI mode.
Enabling traffic prioritization replaces the ENI queue with mq+prio.

Enable traffic prioritization

Run the following command to edit the Terway ConfigMap.
```
kubectl edit cm -n kube-system eni-config
```

In data.10-terway.conf, add the enable_network_priority configuration.

apiVersion: v1
data:
  10-terway.conf: |
    {
      "cniVersion": "0.3.1",
      "name": "terway",
      "enable_network_priority": true,
      "type": "terway"
    }

Save the file, then run the following command to restart the Terway pods and apply the changes.
```
kubectl delete -n kube-system pod -l app=terway-eniip
```

Terway supports the following pod annotations to specify the priority of pod packets.

Annotation	Description
k8s.aliyun.com/network-priority: "guaranteed"	The highest priority. Suitable for latency-sensitive workloads.
k8s.aliyun.com/network-priority: "burstable"	Medium priority. Suitable for workloads that require high bandwidth but are not latency-sensitive.
k8s.aliyun.com/network-priority: "best-effort"	The lowest priority. Suitable for general workloads.