Configure QoS for pods - Container Service for Kubernetes

Online applications are sensitive to network latency while offline applications consume large amounts of bandwidth resources. If you run both types of applications without limits, unexpected traffic spikes may overwhelm your online applications and adversely affect the entire network. To avoid this issue, you can configure quality of service (QoS) for pods. This topic describes the pod annotations defined by Kubernetes and the relevant bandwidth limits. This topic also describes how to configure QoS classes.

Bandwidth limits

You can use the Terway network plug-in to control the pod network. The following table describes the pod annotations that you can use to specify the inbound and outbound bandwidth of pods.

Annotation	Description
kubernetes.io/ingress-bandwidth: 10M	Ingress banwidth
kubernetes.io/egress-bandwidth: 10M	Egress banwidth

The following table describes features supported in different Terway modes for limiting bandwidth.

Terway mode	Egress traffic shaping	Ingress traffic shaping
VPC mode (legacy)
One ENI for multi-pod
One ENI for multi-pod + IPVLAN eBRF

In the One ENI for multi-pod + IPVLAN eBRF mode, pods on nodes that use a kernel version later than 5.10 use Earliest Departure Time (EDT) to limit bandwidth. In other modes, pods use TBF Qdisc to limit bandwidth.

Use pod annotations defined by Kubernetes

In Container Service for Kubernetes (ACK) clusters that use Terway V1.3.0 or later, you can use pod annotations defined by Kubernetes to specify the inbound and outbound bandwidth of pods.
In ACK clusters that use a Terway version earlier than V1.3.0, you can use only the pod annotations defined by Alibaba Cloud to limit the inbound and outbound bandwidth of pods. This following table describes the pod annotations.
Annotation
Description
k8s.aliyun.com/ingress-bandwidth: 10M
Ingress banwidth
k8s.aliyun.com/egress-bandwidth: 10M
Egress banwidth
The pod annotations defined by Alibaba Cloud are deprecated and will soon be removed from ACK. If you want to limit bandwidth, use the pod annotations defined by Kubernetes.
For ACK clusters that use a Terway version earlier than V1.3.0, you need to modify the ConfigMap of Terway to enable the pod annotations defined by Kubernetes.
1. Run the following command to modify the ConfigMap of Terway:
```
kubectl edit cm -n kube-system eni-config
```
2. Add capabilities to the data.10-terway.conf section.
```
apiVersion: v1
data:
  10-terway.conf: |
    {
      "cniVersion": "0.3.1",
      "name": "terway",
      "capabilities": {"bandwidth": true}, #Add the setting to enable bandwidth management.
      "type": "terway"
    }
```
3. Press Esc. Then, enter :wq! and press Enter to save the ConfigMap and exit.
4. Run the following command to recreate the Terway pod for the modified ConfigMap to take effect:
```
kubectl delete -n kube-system pod -l app=terway-eniip
```

Configure QoS classes

By default, the Elastic Network Interface (ENI) queue follows the first-in, first-out (FIFO) principle when processing data packets. This allows you to prioritize the data packets of latency-sensitive applications. By default, traffic prioritization is disabled. You need to manually enable this feature.

The following table describes the pod annotations that can be used to configure QoS classes for pods when Terway is used.

Annotation	Description
k8s.aliyun.com/network-priority: "guaranteed"	Latency-sensitive applications
k8s.aliyun.com/network-priority: "best-effort"	Regular applications
k8s.aliyun.com/network-priority: "burstable"	Applications that consume large amounts of bandwidth

Perform the following steps to enable traffic prioritization.

Run the following command to modify the ConfigMap of Terway:
```
kubectl edit cm -n kube-system eni-config
```

Add enable_network_priority to the data.10-terway.conf section to enable traffic prioritization.

apiVersion: v1
data:
  10-terway.conf: |
    {
      "cniVersion": "0.3.1",
      "name": "terway",
      "enable_network_priority": true, #Add the setting to enable traffic prioritization.
      "type": "terway"
    }

After you modify the ConfigMap, press Esc. Then, enter :wq! and press Enter to save the ConfigMap and exit.
Recreate the Terway pod for the modified ConfigMap to take effect.
```
kubectl delete -n kube-system pod -l app=terway-eniip
```
Important
After traffic prioritization is enabled, the ENI queue is replaced by mq+prio.

Annotation	Description
k8s.aliyun.com/ingress-bandwidth: 10M	Ingress banwidth
k8s.aliyun.com/egress-bandwidth: 10M	Egress banwidth