The ack-koordinator component allows you to dynamically modify the memory parameters of a pod. You can use ack-koordinator to modify the CPU parameters, memory parameters, and disk IOPS limit for a pod. This topic describes how to use ack-koordinator to dynamically modify the resource parameters of a pod.
Prerequisites
A kubectl client is connected to the cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
ack-koordinator (FKA ack-slo-manager) 0.5.0 or later is installed. For more information about how to install ack-koordinator, see ack-koordinator (ack-slo-manager).
Noteack-koordinator is upgraded and optimized based on resource-controller. If you use resource-controller, you must uninstall it before you install ack-koordinator. For more information, see Uninstall resource-controller.
Background information
If you want to modify container parameters for a pod that is running in a Kubernetes cluster, you must modify the PodSpec parameter and submit the change. However, the pod is deleted and recreated after you submit the change. ack-koordinator allows you to dynamically modify the resource parameters of a pod without the need to restart the pod.
Scenarios
This feature is suitable for scenarios where temporary adjustment is required. For example, the memory usage of a pod increases. In this case, you need to increase the memory limit of the pod to avoid triggering the out of memory (OOM) killer. To perform common O&M operations, we recommend that you use the relevant features that are provided by Container Service for Kubernetes (ACK). For more information, see CPU Burst, Topology-aware CPU scheduling, and Resource profiling.
Modify the memory limit
To dynamically modify the memory limit of a pod by using cgroups, perform the following steps:
You can modify the memory limit only in Kubernetes 1.22 and earlier. Other resource parameters can be modified in all Kubernetes versions.
If you modify the CPU limit to meet common requirements, we recommend that you use the CPU Burst feature. This feature can automatically modify the CPU limit of the pod. For more information, see CPU Burst. If you want to dynamically modify the CPU limit of the pod, see Upgrade from resource-controller to ack-koordinator.
Create a pod-demo.yaml file with the following YAML template:
apiVersion: v1 kind: Pod metadata: name: pod-demo spec: containers: - name: pod-demo image: polinux/stress resources: requests: cpu: 1 memory: "50Mi" limits: cpu: 1 memory: "1Gi" # Set the memory limit to 1 GB. command: ["stress"] args: ["--vm", "1", "--vm-bytes", "256M", "-c", "2", "--vm-hang", "1"]
Run the following command to deploy the pod-demo pod in the cluster:
kubectl apply -f pod-demo.yaml
Run the following command to query the original memory limit of the container:
# The actual path consists of the UID of the pod and the ID of the container. cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf44b779_41d8_43d5_a0d8_8a7a0b17****.slice/memory.limit_in_bytes
Expected output:
# In this example, 1073741824 is returned, which is the result of 1 × 1024 × 1024 × 1024. This indicates that the original memory limit of the container is 1 GB. 1073741824
The output shows that the original memory limit of the container is 1 GB, which is the same as the value of the
spec.containers.resources.limits.memory
parameter in the YAML file that you created in Step 1.Create a cgroups-sample.yaml file with the following YAML template:
The template is used to modify the memory limit of the container.
apiVersion: resources.alibabacloud.com/v1alpha1 kind: Cgroups metadata: name: cgroups-sample spec: pod: name: pod-demo namespace: default containers: - name: pod-demo memory: 5Gi # Change the memory limit of the pod to 5 GB.
Run the following command to deploy the cgroups-sample object in the cluster.
kubectl apply -f cgroups-sample.yaml
Run the following command to query the new memory limit of the container after you submit the change:
# The actual path consists of the UID of the pod and the ID of the container. cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf44b779_41d8_43d5_a0d8_8a7a0b17****.slice/memory.limit_in_bytes
Expected output:
# In this example, 5368709120 is returned, which is the result of 5 × 1024 × 1024 × 1024. This indicates that the new memory limit of the container is 5 GB. 5368709120
The output shows that the memory limit of the container is 5 GB, which is the same as the value of the
spec.pod.containers.memory
parameter in the YAML file that you created in Step 4.Run the following command to query the status of the pod:
kubectl describe pod pod-demo
Expected output:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 13m default-scheduler Successfully assigned default/pod-demo to cn-zhangjiakou.192.168.3.238 Normal Pulling 13m kubelet, cn-zhangjiakou.192.168.3.238 Pulling image "polinux/stress" Normal Pulled 13m kubelet, cn-zhangjiakou.192.168.3.238 Successfully pulled image "polinux/stress"
The output shows that the pod runs as normal and no restart
events
are generated.
Change the vCores that are bound to a pod
You can use cgroups to bind vCores to a pod by specifying the serial numbers of vCores. To change the vCores that are bound to a pod, perform the following steps:
In common cases, we recommend that you use topology-aware CPU scheduling to manage CPU resources for CPU-sensitive workloads. For more information, see Topology-aware CPU scheduling.
Create a pod-cpuset-demo.yaml file with the following YAML template:
apiVersion: v1 kind: Pod metadata: name: pod-cpuset-demo spec: containers: - name: pod-cpuset-demo image: polinux/stress resources: requests: memory: "50Mi" limits: memory: "1000Mi" cpu: 0.5 command: ["stress"] args: ["--vm", "1", "--vm-bytes", "556M", "-c", "2", "--vm-hang", "1"]
Run the following command to deploy the pod-cpuset-demo object in the cluster:
kubectl apply -f pod-cpuset-demo.yaml
Run the following command to query the vCores that are bound to the container:
cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b79bee_eb2a_4b67_befe_51c270f8****.slice/cri-containerd-aba883f8b3ae696e99c3a920a578e3649fa957c51522f3fb00ca943dc2c7****.scope/cpuset.cpus
Expected output:
0-31
The output shows that the serial numbers of the vCores that can be used by the container range from 0 to 31 before you bind vCores to the container.
Create a cgroups-sample-cpusetpod.yaml file with the following YAML template.
The YAML file specifies the vCores that are bound to the pods.
apiVersion: resources.alibabacloud.com/v1alpha1 kind: Cgroups metadata: name: cgroups-sample-cpusetpod spec: pod: name: pod-cpuset-demo namespace: default containers: - name: pod-cpuset-demo cpuset-cpus: 2-3 # Bind vCore 2 and vCore 3 to the pod.
Run the following command to deploy the cgroups-sample-cpusetpod object in the cluster:
kubectl apply -f cgroups-sample-cpusetpod.yaml
Run the following command to query the vCores that are bound to the container after you submit the change:
# The actual path consists of the UID of the pod and the ID of the container. cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b79bee_eb2a_4b67_befe_51c270f8****.slice/cri-containerd-aba883f8b3ae696e99c3a920a578e3649fa957c51522f3fb00ca943dc2c7****.scope/cpuset.cpus
Expected output:
2-3
The output shows that vCore 2 and vCore 3 are bound to the container. The vCores that are bound to the container are the same as the vCores that are specified in the
spec.pod.containers.cpuset-cpus
parameter in the YAML file that you created in Step 4.
Modify Deployment-level parameters
To modify Deployment-level parameters by using cgroups, perform the following steps:
Create a go-demo.yaml file with the following YAML template.
The Deployment creates two pods that run a stress testing program. Each pod requests 0.5 vCores.
apiVersion: apps/v1 kind: Deployment metadata: name: go-demo labels: app: go-demo spec: replicas: 2 selector: matchLabels: app: go-demo template: metadata: labels: app: go-demo spec: containers: - name: go-demo image: polinux/stress command: ["stress"] args: ["--vm", "1", "--vm-bytes", "556M", "-c", "1", "--vm-hang", "1"] imagePullPolicy: Always resources: requests: cpu: 0.5 limits: cpu: 0.5
Run the following command to deploy the go-demo Deployment in the cluster:
kubectl apply -f go-demo.yaml
Create a cgroups-cpuset-sample.yaml file with the following YAML template.
The YAML file specifies the vCores that are bound to the pods.
apiVersion: resources.alibabacloud.com/v1alpha1 kind: Cgroups metadata: name: cgroups-cpuset-sample spec: deployment: name: go-demo namespace: default containers: - name: go-demo cpuset-cpus: 2,3 # Bind vCore 2 and vCore 3 to the pods.
Run the following command to deploy the cgroups-cpuset-sample Deployment in the cluster:
kubectl apply -f cgroups-cpuset-sample.yaml
Run the following command to query the vCores that are bound to the container after you submit the change:
# The actual path consists of the UID of the pod and the ID of the container. cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06de7408_346a_4d00_ba25_02833b6c****.slice/cri-containerd-733a0dc93480eb47ac6c5abfade5c22ed41639958e3d304ca1f85959edc3****.scope/cpuset.cpus
Expected output:
2-3
The output shows that vCore 2 and vCore 3 are bound to the containers. The vCores that are bound to the containers are the same as the vCores that are specified in the
spec.deployment.containers.cpuset-cpus
parameter in the YAML file that you created in Step 3.
Modify the disk IOPS
If you want to modify the disk IOPS for a pod, you must use Alibaba Cloud Linux as the operating system of the worker node that you want to manage. To modify the disk IOPS for a pod, perform the following steps:
If you specify a blkio limit in cgoup v1, the OS kernel only limits the direct IOPS of a pod. The OS kernel does not limit the buffered IOPS of a pod. To limit the buffered IOPS of a pod, use cgroup v2 or enable the cgroup writeback feature for Alibaba Cloud Linux. For more information, see Enable the cgroup writeback feature.
Create an I/O-intensive application with the following YAML template.
Mount the host directory /mnt to the pod. The device name of the corresponding disk is /dev/vda1.
apiVersion: apps/v1 kind: Deployment metadata: name: fio-demo labels: app: fio-demo spec: selector: matchLabels: app: fio-demo template: metadata: labels: app: fio-demo spec: containers: - name: fio-demo image: registry.cn-zhangjiakou.aliyuncs.com/acs/fio-for-slo-test:v0.1 command: ["sh", "-c"] # Use Fio to perform write stress tests on the disk. args: ["fio -filename=/data/test -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=12000 -group_reporting -name=mytest"] volumeMounts: - name: pvc mountPath: /data # The disk volume is mounted to the /data path. volumes: - name: pvc hostPath: path: /mnt
Run the following command to deploy the fio-demo Deployment in the cluster:
kubectl apply -f fio-demo.yaml
Deploy a Cgroups object that is used to control the disk IOPS to limit the throughput of the pod.
Create a cgroups-sample-fio.yaml file with the following YAML template.
The YAML file specifies the IOPS limit in bit/s of the /dev/vda1 device.
apiVersion: resources.alibabacloud.com/v1alpha1 kind: Cgroups metadata: name: cgroups-sample-fio spec: deployment: name: fio-demo namespace: default containers: - name: fio-demo blkio: # The IOPS limit in bit/s. Example: 1048576, 2097152, or 3145728. device_write_bps: [{device: "/dev/vda1", value: "1048576"}]
Run the following command to query the disk IOPS limit after you submit the change:
# The actual path consists of the UID of the pod and the ID of the container. cat /sys/fs/cgroup/blkio/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0840adda_bc26_4870_adba_f193cd00****.slice/cri-containerd-9ea6cc97a6de902d941199db2fcda872ddd543485f5f987498e40cd706dc****.scope/blkio.throttle.write_bps_device
Expected output:
253:0 1048576
The output shows that the IOPS limit of the disk is
1048576
bit/s.
View the monitoring data of the node.
The figures show that the disk IOPS limit is changed to the value that is specified in the
device_write_bps
parameter of the YAML file that you created in Step 3. The pod is not restarted after you submit the change.