Container Service for Kubernetes:CPU Burst

Billing rules

No fee is charged when you install and use the ack-koordinator component. However, fees may be charged in the following scenarios:

• ack-koordinator is an non-managed component that occupies worker node resources after it is installed. You can specify the amount of resources requested by each module when you install the component.

• By default, ack-koordinator exposes the monitoring metrics of features such as resource profiling and fine-grained scheduling as Prometheus metrics. If you enable Prometheus metrics for ack-koordinator and use Managed Service for Prometheus, these metrics are considered as custom metrics and fees are charged for these metrics. The fee depends on factors such as the size of your cluster and the number of applications. Before you enable Prometheus metrics, we recommend that you read the Billing topic of Managed Service for Prometheus to learn the free quota and billing rules of custom metrics. For more information about how to monitor and manage resource usage see Query the amount of observable data and bills.

Use scenarios

CPU Burst is suitable in the following scenarios:

• CPU throttling is triggered for an application though the CPU usage of the application is less than the CPU limit of the application. As a result, the performance of the application is degraded. You can enable CPU Burst to resolve this issue and improve the performance of the application.

• The CPU usage during an application startup is higher than the CPU usage after the application is started. You can enable CPU Burst to meet the CPU requirements during an application startup. This way, you do not need to specify an excessively high CPU request for the application, which reduces resource waste.

How it works

Kubernetes allows you to specify resource limits to manage resource allocation. For time-sharing resources such as CPUs, if you specify a CPU limit for a container, the OS limits the amount of CPU resources that can be used by the container within a specific time period. For example, you set the CPU limit of a container to 2. The OS kernel limits the CPU time slices that the container can use to 200 milliseconds within each 100-millisecond period.

CPU utilization is a key metric that is used to evaluate the performance of a container. In most cases, the CPU limit is specified based on CPU utilization. CPU utilization on a per-millisecond basis shows more spikes than on a per-second basis. If the CPU utilization of a container reaches the limit within a 100-millisecond period, CPU throttling is enforced by the OS kernel and threads in the container are suspended for the rest of the time period, as shown in the following figure.

The following figure shows the thread allocation of a web application container that runs on a node with four vCores. The CPU limit of the container is set to 2. The overall CPU utilization within the last second is low. However, Thread 2 cannot be resumed until the third 100-millisecond period starts because CPU throttling is enforced somewhere in the second 100-millisecond period. This increases the response time (RT) and causes long-tail latency problems in containers.

Alibaba Cloud Linux provides the CPU Burst feature. CPU Burst allows a container to accumulate CPU time slices when the container is idle. The container can use the accumulated CPU time slices to burst above the CPU limit when CPU utilization spikes. This improves performance and reduces the RT of the container. For more information, see Enable the CPU burst feature for cgroup v1.

For kernel versions that do not support CPU Burst, ack-koordinator detects CPU throttling events and dynamically adjusts the CPU limit to achieve the same effect as CPU Burst.

The preceding Completely Fair Scheduler (CFS) quota adjustment policy can be used to handle CPU usage spikes For example, when traffic spikes occur, ack-koordinator can eliminate CPU bottlenecks within a few seconds, while ensuring a proper number of workloads on the node.

The CPU Burst feature supported by the kernel of Alibaba Cloud Linux handles CPU usage spikes at a faster rate. We recommend that you enable the CPU Burst feature provided by the kernel of Alibaba Cloud Linux for latency-sensitive applications.

How to use CPU Burst

Use an annotation to enable CPU Burst

annotations:
  # Set the value to auto to enable CPU Burst for the pod. 
  koordinator.sh/cpuBurst: '{"policy": "auto"}'
  # Set the value to none to disable CPU Burst for the pod. 
  #koordinator.sh/cpuBurst: '{"policy": "none"}'

Use a ConfigMap to enable CPU Burst for all pods in a cluster

1. Create a file named configmap.yaml based on the following sample ConfigMap to enable CPU Burst for all pods in a cluster:

   apiVersion: v1
   data:
     cpu-burst-config: '{"clusterStrategy": {"policy": "auto"}}'
     #cpu-burst-config: '{"clusterStrategy": {"policy": "cpuBurstOnly"}}'
     #cpu-burst-config: '{"clusterStrategy": {"policy": "none"}}'
   kind: ConfigMap
   metadata:
     name: ack-slo-config
     namespace: kube-system

2. Check whether the ConfigMap named ack-slo-config exists in the kube-system namespace.

   • If the ConfigMap ack-slo-config exists, use the patch method to update the ConfigMap in case other settings in the ConfigMap are affected.

     kubectl patch cm -n kube-system ack-slo-config --patch "$(cat configmap.yaml)

   • If the ConfigMap ack-slo-config does not exist, run the following command to create one:

     kubectl apply -f configmap.yaml

Enable CPU Burst for pods in specified namespaces

apiVersion: v1
kind: ConfigMap
metadata:
  name: ack-slo-pod-config
  namespace: koordinator-system # You need to manually create the namespace during the first time.
data:
  # Enable or disable CPU Burst for pods in specified namespaces. 
  cpu-burst: |
    {
      "enabledNamespaces": ["white-ns"],
      "disabledNamespaces": ["black-ns"]
    }

Advanced configurations of CPU Burst

# Example of the ack-slo-config ConfigMap. 
data:
  cpu-burst-config: |
    {
      "clusterStrategy": {
        "policy": "auto",
        "cpuBurstPercent": 1000,
        "cfsQuotaBurstPercent": 300,
        "sharePoolThresholdPercent": 50,
        "cfsQuotaBurstPeriodSeconds": -1
      }
    }

# Example of pod annotations. 
  koordinator.sh/cpuBurst: '{"policy": "auto", "cpuBurstPercent": 1000, "cfsQuotaBurstPercent": 300, "cfsQuotaBurstPeriodSeconds": -1}'

The following table describes the advanced parameters of CPU Burst.

Verify the effect of CPU Burst

Verification steps

1. Use the following YAML template to create an apache-demo.yaml file:

   To enable CPU Burst for a pod, specify an annotation in the annotations parameter of the metadata section of the pod configuration.

   apiVersion: v1
   kind: Pod
   metadata:
     name: apache-demo
     annotations:
       koordinator.sh/cpuBurst: '{"policy": "auto"}'   # The annotation is used to enable or disable CPU Burst. 
   spec:
     containers:
     - command:
       - httpd
       - -D
       - FOREGROUND
       image: registry.cn-zhangjiakou.aliyuncs.com/acs/apache-2-4-51-for-slo-test:v0.1
       imagePullPolicy: Always
       name: apache
       resources:
         limits:
           cpu: "4"
           memory: 10Gi
         requests:
           cpu: "4"
           memory: 10Gi
     nodeName: $nodeName # Replace nodeName with the actual node name. 
     hostNetwork: False
     restartPolicy: Never
     schedulerName: default-scheduler

2. Run the following command to create an application by using Apache HTTP Server:

   kubectl apply -f apache-demo.yaml

3. Use the wrk2 tool to perform stress tests.

# Download, decompress, and then install the wrk2 package. For more information, visit https://github.com/giltene/wrk2. 
# Gzip compression is enabled in the Apache image to simulate the request processing logic of the server. 
# Run the following command to send requests. Replace the IP address in the command with the IP address of the application. 
./wrk -H "Accept-Encoding: deflate, gzip" -t 2 -c 12 -d 120 --latency --timeout 2s -R 24 http://$target_ip_address:8010/static/file.1m.test

Note

• Replace the IP address in the command with the pod IP address of the Apache application.

• You can modify the -R field to change the number of queries per unit time from the sender.

Analyze the result

The following tables show metrics before and after CPU Burst is enabled for Alibaba Cloud Linux and CentOS.

• The Disabled column shows the metrics when the CPU Burst policy is set to none.

• The Enabled column shows the metrics when the CPU Burst policy is set to auto.

The preceding metrics indicate the following information:

• After CPU Burst is enabled, the P99 latency is greatly reduced.

• After CPU Burst is enabled, CPU throttling is stopped and the average pod CPU utilization remains approximately at the same value.

FAQ

Is the CPU Burst feature that is enabled based on the earlier version of the ack-slo-manager protocol supported after I upgrade ack-slo-manager to ack-koordinator?

The earlier version of the pod protocol requires you to add the alibabacloud.com/cpuBurst annotation. ack-koordinator is fully compatible with the earlier protocol version. You can seamlessly upgrade from ack-slo-manager to ack-koordinator.

The following table describes the compatibilities between ack-koordinator and different types of protocols.