Kubernetes allows you to deploy various types of containerized applications on the same node. This causes applications with different priorities to compete for CPU resources. As a result, the performance of the applications with high priorities cannot be guaranteed. The ack-koordinator component allows you to use quality of service (QoS) classes to guarantee CPU resources for applications with high priorities. This topic describes how to configure the CPU QoS feature for pods.
Prerequisites
- Only Container Service for Kubernetes (ACK) Pro clusters support the dynamic resource overcommitment feature. Fore more information, see Create an ACK Pro cluster.
- ack-koordinator (FKA ack-slo-manager) is installed. Fore more information, see ack-koordinator (FKA ack-slo-manager).
Background information
To fully utilize computing resources, workloads of different priorities are usually deployed on the same node. For example, latency-sensitive (LS) workloads (with high priorities) and best-effort (BE) workloads (with low priorities) can be deployed on the same node. However, this may cause these workloads to compete for computing resources. In Kubernetes, CPU requests and CPU limits are used to control the amount of CPU resources that pods can use. However, pods may still compete for CPU resources. For example, BE pods and LS pods can share CPU cores or vCPU cores. When the loads of the BE pods increase, the performance of the LS pods is compromised. As a result, the response latency of the application that uses the LS pods increases.
To reduce the performance impact on the BE pods in this scenario, you can use the CPU QoS feature provided by the ack-koordinator component to limit the CPU usage of the LS pods. The CPU QoS feature is based on Alibaba Cloud Linux 2. The ack-koordinator component allows you to use the group identity feature available in Alibaba Cloud Linux 2 to configure Linux scheduling priorities for pods. In an environment where both LS pods and BE pods are deployed, you can set the priority of LS pods to high and the priority of BE pods to low to avoid resource contention. The LS pods are prioritized to use the limited CPU resources to ensure the service quality of the corresponding application. For more information, see Group identity feature.
- The wake-up latency of tasks for LS workloads is minimized.
- Waking up tasks for BE workloads does not adversely impact the performance of LS pods.
- Tasks for BE workloads cannot use the simultaneous multithreading (SMT) scheduler to share CPU cores. This further reduces the impact on the performance of LS pods.
Limits
The following table describes the versions of the system components that are required for using the CPU QoS feature.
Component | Required version |
---|---|
Kubernetes | ≥ 1.18 |
ack-koordinator (FKA ack-slo-manager) | ≥ 0.8.0 |
Helm | ≥ 3.0 |
OS | Alibaba Cloud Linux 2 (For more information, see Group identity feature) |
Procedure
- Create a configmap.yaml file based on the following ConfigMap content:
Specify# Example of the ack-slo-config ConfigMap. apiVersion: v1 kind: ConfigMap metadata: name: ack-slo-config namespace: kube-system data: # Enable the CPU QoS feature. resource-qos-config: | { "clusterStrategy": { "lsClass": { "cpuQOS": { "enable": true, "groupIdentity": 2 } }, "beClass": { "cpuQOS": { "enable": true, "groupIdentity": -1 } } } }
lsClass
andbeClass
to assign the LS and BE classes to different pods.cpuQOS
includes the CPU QoS parameters. The following table describes the parameters.Configuration item Parameter Valid values Description enable
Boolean - true
- false
- true: enables the CPU QoS feature for all containers in the cluster.
- false: disables the CPU QoS feature for all containers in the cluster.
groupIdentity
Int -1~2 Specify group identities for CPU scheduling. By default, the group identity of LS pods is 2 and the group identity of BE pods is -1. A value of 0 indicates that no group identity is assigned. A greater
group identity
value indicates a higher priority in CPU scheduling. For example, you can setcpu.bvt_warp_ns=2
for LS pods and setcpu.bvt_warp_ns=-1
for BE pods because the priority of LS pods is higher than that of BE pods. For more information, see Group identity feature.Note Ifkoordinator.sh/qosClass
is not specified for a pod, the ack-koordinator component configures the pod based on the original QoS class of the pod. The component uses the BE settings in the preceding ConfigMap if the original QoS class is BE. The component uses the LS settings in the preceding ConfigMap if the original QoS class is not BE - Check whether a ConfigMap named
ack-slo-config
exists in thekube-system
namespace.- If a ConfigMap named
ack-slo-config
exists, we commend that you run the kubectl patch command to update the ConfigMap. This avoids changing other settings in the ConfigMap.kubectl patch cm -n kube-system ack-slo-config --patch "$(cat configmap.yaml)"
- If no ConfigMap named
ack-slo-config
exists, run the kubectl patch command to create a ConfigMap named ack-slo-config:kubectl apply -f configmap.yaml
- If a ConfigMap named
- Create a file named ls-pod-demo.yaml based on the following YAML content:
apiVersion: v1 kind: Pod metadata: name: ls-pod-demo labels: koordinator.sh/qosClass: 'LS' # Set the QoS class of the pod to LS. spec: containers: - command: - httpd - -D - FOREGROUND image: registry.cn-zhangjiakou.aliyuncs.com/acs/apache-2-4-51-for-slo-test:v0.1 imagePullPolicy: Always name: apache resources: limits: cpu: "4" memory: 10Gi requests: cpu: "4" memory: 10Gi restartPolicy: Never schedulerName: default-scheduler
- Run the following command to deploy the ls-pod-demo pod in the cluster:
kubectl apply -f ls-pod-demo.yaml
- Run the following command to check whether the CPU group identity of the LS pod in the control group (cgroup) of the node takes effect:
cat /sys/fs/cgroup/cpu/kubepods.slice/kubepods-pod1c20f2ad****.slice/cpu.bvt_warp_ns
Expected output:#The group identity of the LS pod is 2 (high priority). 2
- Create a file named be-pod-demo.yaml based on the following content:
apiVersion: v1 kind: Pod metadata: name: be-pod-demo labels: koordinator.sh/qosClass: 'BE' # Set the QoS class of the pod to BE. spec: containers: - args: - '-c' - '1' - '--vm' - '1' command: - stress image: polinux/stress imagePullPolicy: Always name: stress restartPolicy: Always schedulerName: default-scheduler
- Run the following command to deploy the be-pod-demo pod in the cluster:
kubectl apply -f be-pod-demo.yaml
- Run the following command to check whether the CPU group identity of the BE pod in the cgroup of the node takes effect:
cat /sys/fs/cgroup/cpu/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b6e96c8****.slice/cpu.bvt_warp_ns
Expected output:
The output shows that the priority of the LS pod is high and the priority of the BE pod is low. CPU resources are preferably scheduled to the LS pod to ensure the service quality.#The group identity of the BE pod is -1 (low priority). -1
FAQ
Is the CPU QoS feature that is enabled based on the earlier version of the ack-slo-manager protocol supported after I upgrade from ack-slo-manager to ack-koordinator?
In an earlier version (≤ 0.8.0) of the ack-slo-manager protocol, the alibabacloud.com/qosClass
pod annotation is used.
ack-koordinator is compatible with the earlier versions of the ack-slo-manager protocol. You can seamlessly upgrade from ack-slo-manager to ack-koordinator. ack-koordinator is compatible with the earlier protocol versions until July 30, 2023. We recommend that you upgrade the resource parameters in an earlier protocol version to the latest version.
ack-koordinator version | alibabacloud.com protocol | koordinator.sh protocol |
---|---|---|
≥ 0.5.2 and < 0.8.0 | Supported | Not supported |
≥ 0.8.0 | Supported | Supported |