Container Service for Kubernetes:Dynamically modify pod resource parameters

How it works

ACK uses a custom resource definition (CRD) of kind: Cgroups to pass resource changes to ack-koordinator. When you apply a Cgroups resource, koordlet on the node writes new values directly to cgroup files — bypassing the Kubernetes scheduler and kubelet reconciliation loop. The pod's PodSpec remains unchanged; only cgroup values on the node are updated.

Use spec.pod to target a specific pod, or spec.deployment to target all pods in a Deployment.

Prerequisites

Ensure that:

• kubectl is connected to the ACK cluster.

• ack-koordinator 0.5.0 or later is installed.

Billing

ack-koordinator is free to install and use. Additional charges may apply in these situations:

• Worker node resources: ack-koordinator is an unmanaged add-on that consumes resources on worker nodes after installation. Specify the resource requests for each module when you install the add-on.

• Prometheus monitoring metrics: If you enable the Enable Prometheus monitoring metrics for ACK-Koordinator option and use Managed Service for Prometheus, the metrics are billed as custom metrics. Charges depend on cluster size and number of applications. Review the Prometheus instance billing documentation and use usage query to monitor consumption before enabling this feature.

Limitations

Modify the memory limit

Raise a pod's memory limit to prevent OOM kills without restarting it. This example raises the limit from 1 GiB to 5 GiB.

1. Create pod-demo.yaml with the following content.

   apiVersion: v1
   kind: Pod
   metadata:
     name: pod-demo
   spec:
     containers:
     - name: pod-demo
       image: registry-cn-beijing.ack.aliyuncs.com/acs/stress:v1.0.4
       resources:
         requests:
           cpu: 1
           memory: "50Mi"
         limits:
           cpu: 1
           memory: "1Gi" # Initial memory limit: 1 GiB
       command: ["stress"]
       args: ["--vm", "1", "--vm-bytes", "256M", "-c", "2", "--vm-hang", "1"]

2. Deploy the pod.

   kubectl apply -f pod-demo.yaml

3. Verify the initial memory limit. The cgroup path is constructed from the pod UID and container ID.

   cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf44b779_41d8_43d5_a0d8_8a7a0b17****.slice/memory.limit_in_bytes

   Expected output:

   1073741824

   1073741824 = 1 GiB, matching spec.containers.resources.limits.memory in the pod definition.

4. Create cgroups-sample.yaml to set the new memory limit.

   apiVersion: resources.alibabacloud.com/v1alpha1
   kind: Cgroups
   metadata:
     name: cgroups-sample
   spec:
     pod:
       name: pod-demo
       namespace: default
       containers:
       - name: pod-demo
         memory: 5Gi  # New memory limit: 5 GiB

5. Apply the Cgroups resource.

   kubectl apply -f cgroups-sample.yaml

6. Verify the updated memory limit.

   cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf44b779_41d8_43d5_a0d8_8a7a0b17****.slice/memory.limit_in_bytes

   Expected output:

   5368709120

   5368709120 = 5 GiB, matching spec.pod.containers.memory in the Cgroups resource.

7. Confirm that the pod was not restarted.

   kubectl describe pod pod-demo

   Confirm no restart events in the Events section — only original scheduling and startup events:

   Events:
     Type    Reason          Age   From               Message
     ----    ------          ----  ----               -------
     Normal  Scheduled       36m   default-scheduler  Successfully assigned default/pod-demo to cn-hangzhou.192.168.0.50
     Normal  AllocIPSucceed  36m   terway-daemon      Alloc IP 192.XX.XX.51/24 took 4.490542543s
     Normal  Pulling         36m   kubelet            Pulling image "registry-cn-beijing.ack.aliyuncs.com/acs/stress:v1.0.4"
     Normal  Pulled          36m   kubelet            Successfully pulled image "registry-cn-beijing.ack.aliyuncs.com/acs/stress:v1.0.4" in 2.204s (2.204s including waiting). Image size: 7755078 bytes.
     Normal  Created         36m   kubelet            Created container pod-demo
     Normal  Started         36m   kubelet            Started container pod-demo

   No restart events confirm the memory limit was updated in place.

Modify the CPU core binding scope

Bind a pod to specific CPU cores for stricter resource isolation. This example restricts a pod from all 32 cores (0–31) to cores 2–3.

1. Create pod-cpuset-demo.yaml with the following content.

   apiVersion: v1
   kind: Pod
   metadata:
     name: pod-cpuset-demo
   spec:
     containers:
     - name: pod-cpuset-demo
       image: registry-cn-beijing.ack.aliyuncs.com/acs/stress:v1.0.4
       resources:
         requests:
           memory: "50Mi"
         limits:
           memory: "1000Mi"
           cpu: 0.5
       command: ["stress"]
       args: ["--vm", "1", "--vm-bytes", "556M", "-c", "2", "--vm-hang", "1"]

2. Deploy the pod.

   kubectl apply -f pod-cpuset-demo.yaml

3. Check the current CPU core binding. The path is constructed from the pod UID and container ID.

   cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b79bee_eb2a_4b67_befe_51c270f8****.slice/cri-containerd-aba883f8b3ae696e99c3a920a578e3649fa957c51522f3fb00ca943dc2c7****.scope/cpuset.cpus

   Expected output:

   0-31

   0-31 means the container can use all 32 CPU cores.

4. Create cgroups-sample-cpusetpod.yaml to set the CPU binding.

   apiVersion: resources.alibabacloud.com/v1alpha1
   kind: Cgroups
   metadata:
     name: cgroups-sample-cpusetpod
   spec:
     pod:
       name: pod-cpuset-demo
       namespace: default
       containers:
       - name: pod-cpuset-demo
         cpuset-cpus: 2-3  # Restrict the pod to CPU cores 2 and 3

5. Apply the Cgroups resource.

   kubectl apply -f cgroups-sample-cpusetpod.yaml

6. Verify the updated CPU binding.

   cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b79bee_eb2a_4b67_befe_51c270f8****.slice/cri-containerd-aba883f8b3ae696e99c3a920a578e3649fa957c51522f3fb00ca943dc2c7****.scope/cpuset.cpus

   Expected output:

   2-3

   The container is bound to cores 2–3, matching spec.pod.containers.cpuset-cpus in the Cgroups resource.

7. Confirm that the pod was not restarted.

   kubectl describe pod pod-cpuset-demo

   The Events section includes a CPUSetBind event from koordlet but no restart events:

   Events:
     Type    Reason          Age   From               Message
     ----    ------          ----  ----               -------
     Normal  Scheduled       7m7s  default-scheduler  Successfully assigned default/pod-cpuset-demo to cn-hangzhou.192.XX.XX.50
     Normal  AllocIPSucceed  7m5s  terway-daemon      Alloc IP 192.XX.XX.56/24 took 2.060752512s
     Normal  Pulled          7m5s  kubelet            Container image "registry-cn-beijing.ack.aliyuncs.com/acs/stress:v1.0.4" already present on machine
     Normal  Created         7m5s  kubelet            Created container pod-cpuset-demo
     Normal  Started         7m5s  kubelet            Started container pod-cpuset-demo
     Normal  CPUSetBind      84s   koordlet           set cpuset 2-3 to container pod-cpuset-demo success

Modify disk I/O parameters

This example deploys an fio workload and limits its write throughput via a cgroup file.

1. Create fio-demo.yaml with the following content. Host directory /mnt is mounted at /data in the pod, mapped to /dev/vda1.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: fio-demo
     labels:
       app: fio-demo
   spec:
     selector:
       matchLabels:
         app: fio-demo
     template:
       metadata:
         labels:
           app: fio-demo
       spec:
         containers:
         - name: fio-demo
           image: registry.cn-zhangjiakou.aliyuncs.com/acs/fio-for-slo-test:v0.1
           command: ["sh", "-c"]
           # Run a sequential write test on disk I/O using fio
           args: ["fio -filename=/data/test -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=12000 -group_reporting -name=mytest"]
           volumeMounts:
             - name: pvc
               mountPath: /data
         volumes:
           - name: pvc
             hostPath:
               path: /mnt

2. Deploy the application.

   kubectl apply -f fio-demo.yaml

3. Limit the write throughput using a cgroup file.

   1. Create cgroups-sample-fio.yaml to set a bytes-per-second (BPS) write limit on /dev/vda1.

      apiVersion: resources.alibabacloud.com/v1alpha1
      kind: Cgroups
      metadata:
        name: cgroups-sample-fio
      spec:
        deployment:
          name: fio-demo
          namespace: default
          containers:
          - name: fio-demo
            blkio:
              # BPS limit in bytes per second (e.g., 1048576 = 1 MiB/s)
              device_write_bps: [{device: "/dev/vda1", value: "1048576"}]

   2. Apply the Cgroups resource.

      kubectl apply -f cgroups-sample-fio.yaml

   3. Verify the updated disk I/O limit. The path is constructed from the pod UID and container ID.

      cat /sys/fs/cgroup/blkio/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0840adda_bc26_4870_adba_f193cd00****.slice/cri-containerd-9ea6cc97a6de902d941199db2fcda872ddd543485f5f987498e40cd706dc****.scope/blkio.throttle.write_bps_device

      Expected output:

      253:0 1048576

      Write BPS limit is 1048576 (1 MiB/s) for device 253:0 (/dev/vda1). The pod was not restarted.

4. To view disk monitoring data in Prometheus, go to the console and choose Operations > Prometheus Monitoring. On the Application Monitoring tab, filter for the sample application. See Connect to and configure Alibaba Cloud Prometheus Monitoring.

Apply changes at the Deployment level

All procedures above also work at the Deployment level — use spec.deployment instead of spec.pod. This example applies CPU core binding to a Deployment.

1. Create go-demo.yaml with the following content. This Deployment runs two replicas with 0.5 CPU each.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: go-demo
     labels:
       app: go-demo
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: go-demo
     template:
       metadata:
         labels:
           app: go-demo
       spec:
         containers:
         - name: go-demo
           image: registry-cn-beijing.ack.aliyuncs.com/acs/stress:v1.0.4
           command: ["stress"]
           args: ["--vm", "1", "--vm-bytes", "556M", "-c", "1", "--vm-hang", "1"]
           imagePullPolicy: Always
           resources:
             requests:
               cpu: 0.5
             limits:
               cpu: 0.5

2. Deploy the application.

   kubectl apply -f go-demo.yaml

3. Create cgroups-cpuset-sample.yaml to bind the Deployment's pods to specific CPU cores.

   apiVersion: resources.alibabacloud.com/v1alpha1
   kind: Cgroups
   metadata:
     name: cgroups-cpuset-sample
   spec:
     deployment: # Targets a Deployment, not a single pod
       name: go-demo
       namespace: default
       containers:
       - name: go-demo
         cpuset-cpus: 2,3 # Bind to CPU cores 2 and 3

4. Apply the Cgroups resource.

   kubectl apply -f cgroups-cpuset-sample.yaml

5. Verify the CPU binding for one of the pods. The path is constructed from the pod UID and container ID.

   cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06de7408_346a_4d00_ba25_02833b6c****.slice/cri-containerd-733a0dc93480eb47ac6c5abfade5c22ed41639958e3d304ca1f85959edc3****.scope/cpuset.cpus

   Expected output:

   2-3

   The container is bound to cores 2–3, matching spec.deployment.containers.cpuset-cpus in the Cgroups resource.

Next steps

• CPU Burst: Containers accumulate unused CPU time slices and spend them during traffic spikes, reducing latency and improving quality of service. See Enable the CPU Burst policy.

• CPU topology-aware scheduling: Pin pods to specific CPU cores at scheduling time to eliminate CPU context-switching overhead and cross-NUMA memory access. See Enable CPU topology-aware scheduling.

• Dynamic resource overselling: Reclaim allocated-but-unused resources and make them available to lower-priority workloads. Combine with single-node QoS policies to avoid performance interference between applications. See Enable dynamic resource overselling.

• Resource profiling: Analyze historical usage data to get right-sizing recommendations for container requests and limits. See Resource profiling.