The ack-slo-manager component allows you to dynamically modify the memory parameters of a pod. You can use ack-slo-manager to modify the CPU parameters, memory parameters, and the disk I/O limit for a pod. This topic describes how to use ack-slo-manager to dynamically modify the resource parameters of a pod.

Prerequisites

  • A kubectl client is connected to the cluster. For more information, see Connect to ACK clusters by using kubectl.
  • ack-slo-manager is installed and the version is 0.5.0 or later. For more information about how to install ack-slo-manager, see Usage notes.
    Note ack-slo-manager is upgraded and optimized based on resource-controller. You must uninstall resource-controller after you install ack-slo-manager. For more information about how to uninstall a component, see Manage system components.

Background information

If you want to modify container parameters for a pod that is running in a Kubernetes cluster, you must modify the PodSpec parameter and submit the change. However, the pod is deleted and recreated after you submit the change. ack-slo-manager allows you to dynamically modify the resource parameters of a pod without the need to restart the pod.

Scenarios

This feature is suitable for scenarios where temporary adjustment is required. For example, the memory usage of a pod increases. In this case, you need to increase the memory limit of the pod to avoid triggering the out of memory (OOM) killer. To perform common O&M operations, we recommend that you use the relevant features that are provided by Container Service for Kubernetes (ACK). For more information, see CPU Burst, Topology-aware CPU scheduling, and Obtain recommendations on resource specifications.

Modify the memory limit

To dynamically modify the memory limit of a pod by using cgroups, perform the following steps:

Note If you modify the CPU limit to meet common requirements, we recommend that you use the CPU Burst feature. This feature can automatically modify the CPU limit of the pod. For more information, see CPU Burst. If you want to dynamically modify the CPU limit of the pod, Submit a ticket.
  1. Create a pod-demo.yaml file with the following YAML template:
    apiVersion: v1
    kind: Pod
    metadata:
      name: pod-demo
    spec:
      containers:
      - name: pod-demo
        image: polinux/stress
        resources:
          requests:
            cpu: 1
            memory: "50Mi"
          limits:
            cpu: 1 
            memory: "1Gi" # Set the memory limit to 1 GB. 
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "256M", "-c", "2", "--vm-hang", "1"]
  2. Run the following command to deploy the pod-demo pod in the cluster:
    kubectl apply -f pod-demo.yaml
  3. Run the following command to query the original memory limit of the container:
    # The actual path consists of the UID of the pod and the ID of the container. 
    cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf44b779_41d8_43d5_a0d8_8a7a0b17****.slice/memory.limit_in_bytes

    Expected output:

    # In this example, 1073741824 is returned, which is the result of 1 × 1024 × 1024 × 1024. This indicates that the original memory limit of the container is 1 GB. 
    1073741824

    The output shows that the original memory limit of the container is 1 GB, which is the same as the value of the spec.containers.resources.limits.memory parameter in the YAML file that you created in Step 1.

  4. Create a cgroups-sample.yaml file with the following YAML template:
    The template is used to modify the memory limit of the container.
    apiVersion: resources.alibabacloud.com/v1alpha1
    kind: Cgroups
    metadata:
      name: cgroups-sample
    spec:
      pod:
        name: pod-demo
        namespace: default
        containers:
        - name: pod-demo
          memory: 5Gi  # Change the memory limit of the pod to 5 GB. 
  5. Run the following command to deploy the cgroups-sample object in the cluster.
    kubectl apply -f cgroups-sample.yaml
  6. Run the following command to query the new memory limit of the container after you submit the change:
    # The actual path consists of the UID of the pod and the ID of the container. 
    cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf44b779_41d8_43d5_a0d8_8a7a0b17****.slice/memory.limit_in_bytes

    Expected output:

    # In this example, 5368709120 is returned, which is the result of 5 × 1024 × 1024 × 1024. This indicates that the new memory limit of the container is 5 GB.
    5368709120

    The output shows that the memory limit of the container is 5 GB, which is the same as the value of the spec.pod.containers.memory parameter in the YAML file that you created in Step 4.

  7. Run the following command to query the status of the pod:
    kubectl describe pod pod-demo

    Expected output:

    Events:
    Type    Reason            Age   From                                   Message
    ----    ------            ----  ----                                   -------
    Normal  Scheduled         13m   default-scheduler                      Successfully assigned default/pod-demo to cn-zhangjiakou.192.168.3.238
    Normal  Pulling           13m   kubelet, cn-zhangjiakou.192.168.3.238  Pulling image "polinux/stress"
    Normal  Pulled            13m   kubelet, cn-zhangjiakou.192.168.3.238  Successfully pulled image "polinux/stress"

    The output shows that the pod runs as normal and no restart events are generated.

Change the CPU cores that are bound to a pod

You can use cgroups to bind CPU cores to a pod by specifying the serial numbers of CPU cores. To change the CPU cores that are bound to a pod, perform the following steps:

Note In common cases, we recommend that you use topology-aware CPU scheduling to manage CPU resources for CPU-sensitive workloads. For more information, see Topology-aware CPU scheduling.
  1. Create a pod-cpuset-demo.yaml file with the following YAML template:
    apiVersion: v1
    kind: Pod
    metadata:
      name: pod-cpuset-demo
    spec:
      containers:
      - name: pod-cpuset-demo
        image: polinux/stress
        resources:
          requests:
            memory: "50Mi"
          limits:
            memory: "1000Mi"
            cpu: 0.5
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "556M", "-c", "2", "--vm-hang", "1"]
  2. Run the following command to deploy the pod-cpuset-demo object in the cluster:
    kubectl apply -f pod-cpuset-demo.yaml
  3. Run the following command to query the CPU cores that are bound to the container:
    cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b79bee_eb2a_4b67_befe_51c270f8****.slice/cri-containerd-aba883f8b3ae696e99c3a920a578e3649fa957c51522f3fb00ca943dc2c7****.scope/cpuset.cpus

    Expected output:

    0-31

    The output shows that the serial numbers of the CPU cores that can be used by the container range from 0 to 31 before you bind CPU cores to the container.

  4. Create a cgroups-sample-cpusetpod.yaml file with the following YAML template.
    The YAML file specifies the CPU cores that are bound to the pod.
    apiVersion: resources.alibabacloud.com/v1alpha1
    kind: Cgroups
    metadata:
      name: cgroups-sample-cpusetpod
    spec:
      pod:
        name: pod-cpuset-demo
        namespace: default
        containers:
        - name: pod-cpuset-demo
          cpuset-cpus: 2-3  # Bind CPU Core 2 and CPU Core 3 to the pod. 
  5. Run the following command to deploy the cgroups-sample-cpusetpod object in the cluster:
    kubectl apply -f cgroups-sample-cpusetpod.yaml
  6. Run the following command to query the CPU cores that are bound to the container after you submit the change:
    # The actual path consists of the UID of the pod and the ID of the container. 
    cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b79bee_eb2a_4b67_befe_51c270f8****.slice/cri-containerd-aba883f8b3ae696e99c3a920a578e3649fa957c51522f3fb00ca943dc2c7****.scope/cpuset.cpus

    Expected output:

    2-3

    The output shows that CPU Core 2 and CPU Core 3 are bound to the container. The CPU cores that are bound to the container are the same as the CPU cores that are specified in the spec.pod.containers.cpuset-cpus parameter in the YAML file that you created in Step 4.

Modify Deployment-level parameters

To modify Deployment-level parameters by using cgroups, perform the following steps:

  1. Create a go-demo.yaml file with the following YAML template.
    The Deployment creates two pods that run a stress testing program. Each pod requests 0.5 CPU cores.
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: go-demo
      labels:
        app: go-demo
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: go-demo
      template:
        metadata:
          labels:
            app: go-demo
        spec:
          containers:
          - name: go-demo
            image: polinux/stress
            command: ["stress"]
            args: ["--vm", "1", "--vm-bytes", "556M", "-c", "1", "--vm-hang", "1"]
            imagePullPolicy: Always
            resources:
              requests:
                cpu: 0.5
              limits:
                cpu: 0.5
  2. Run the following command to deploy the go-demo Deployment in the cluster:
    kubectl apply -f go-demo.yaml
  3. Create a cgroups-cpuset-sample.yaml file with the following YAML template.
    The YAML file specifies the CPU cores that are bound to the pods.
    apiVersion: resources.alibabacloud.com/v1alpha1
    kind: Cgroups
    metadata:
      name: cgroups-cpuset-sample
    spec:
      deployment:
        name: go-demo
        namespace: default
        containers:
        - name: go-demo
          cpuset-cpus: 2,3 # Bind CPU Core 2 and CPU Core 3 to the pods. 
  4. Run the following command to deploy the cgroups-cpuset-sample Deployment in the cluster:
    kubectl apply -f cgroups-cpuset-sample.yaml
  5. Run the following command to query the CPU cores that are bound to the container after you submit the change:
    # The actual path consists of the UID of the pod and the ID of the container. 
    cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06de7408_346a_4d00_ba25_02833b6c****.slice/cri-containerd-733a0dc93480eb47ac6c5abfade5c22ed41639958e3d304ca1f85959edc3****.scope/cpuset.cpus

    Expected output:

    2-3

    The output shows that CPU Core 2 and CPU Core 3 are bound to the containers. The CPU cores that are bound to the containers are the same as the CPU cores that are specified in the spec.deployment.containers.cpuset-cpus parameter in the YAML file that you created in Step 3.

Modify the disk I/O for a pod

If you want to modify the disk I/O for a pod, you must use Alibaba Cloud Linux as the operating system of the worker node that you want to manage. To modify the disk I/O for a pod, perform the following steps:

  1. Create an I/O-intensive application with the following YAML template.
    Mount the host directory /mnt to the pod. The device name of the corresponding disk is /dev/vda1.
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: fio-demo
      labels:
        app: fio-demo
    spec:
      selector:
        matchLabels:
          app: fio-demo
      template:
        metadata:
          labels:
            app: fio-demo
        spec:
          containers:
          - name: fio-demo
            image: registry.cn-zhangjiakou.aliyuncs.com/acs/fio-for-slo-test:v0.1
            command: ["sh", "-c"]
            # Use Fio to perform write stress tests on the disk. 
            args: ["fio -filename=/data/test -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=12000 -group_reporting -name=mytest"]
            volumeMounts:
              - name: pvc
                mountPath: /data    # The disk volume is mounted to the /data path. 
          volumes:
            - name: pvc
              hostPath:
                path: /mnt
  2. Run the following command to deploy the fio-demo Deployment in the cluster:
    kubectl apply -f fio-demo.yaml
  3. Deploy a Cgroups object that is used to control the disk I/O to limit the throughput of the pod.
    1. Create a cgroups-sample-fio.yaml file with the following YAML template.
      The YAML file specifies the I/O limit in bit/s of the /dev/vda1 device.
      apiVersion: resources.alibabacloud.com/v1alpha1
      kind: Cgroups
      metadata:
        name: cgroups-sample-fio
      spec:
        deployment:
          name: fio-demo
          namespace: default
          containers:
          - name: fio-demo
            blkio:
              # The I/O limit in bit/s. Example: 1048576, 2097152, or 3145728. 
              device_write_bps: [{device: "/dev/vda1", value: "1048576"}]
    2. Run the following command to query the disk I/O limit after you submit the change:
      # The actual path consists of the UID of the pod and the ID of the container. 
      cat /sys/fs/cgroup/blkio/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0840adda_bc26_4870_adba_f193cd00****.slice/cri-containerd-9ea6cc97a6de902d941199db2fcda872ddd543485f5f987498e40cd706dc****.scope/blkio.throttle.write_bps_device
      Expected output:
      253:0 1048576

      The output shows that the I/O limit of the disk is 1048576 bit/s.

  4. View the monitoring data of the node.
    8485The figures show that the disk I/O limit is changed to the value that is specified in the device_write_bps parameter of the YAML file that you created in Step 3. The pod is not restarted after you submit the change.