Container Service for Kubernetes:Enable the CPU Burst performance optimization policy

Benefits

CPU utilization is a key metric for measuring the running status of a container. Cluster administrators usually refer to this metric when they configure the CPU limit of a container. Compared with the commonly used second-level metrics, the CPU utilization of a container at the 100-millisecond level often shows more obvious spikes and reveals more instantaneous fluctuations. Take the following figure as an example: if you measure at the second level (the purple line), the CPU usage is clearly less than 4 cores; if you measure at the 100-millisecond level (the green line), the CPU usage of the container exceeds 4 cores during some periods. In this case, if you set the CPU limit to 4 cores, threads are suspended by the operating system because of CPU throttling.

The following figure shows how CPU resources are allocated to each thread (Thread) of a web service container with a CPU limit of 2 on a 4-core node after the container receives requests (req). The left figure shows the normal case, and the right figure shows the case after CPU Burst is enabled.

In addition to the preceding scenarios, CPU Burst also applies to scenarios in which the CPU resource demand of a container surges. For example, when business traffic suddenly increases, ack-koordinator can quickly resolve CPU resource bottlenecks within seconds while ensuring that the node-wide load stays at a safe level.

Scenarios

The typical scenarios of the CPU Burst feature are as follows:

• Most of the time, the CPU resource usage of a container is lower than the configured CPU limit, but CPU throttling still occurs frequently and affects application performance. Enabling CPU Burst allows the container to use the accumulated time slices during load bursts, which effectively resolves CPU throttling and improves the Quality of Service (QoS) of the application.

• A container application consumes a large amount of CPU resources during the startup and loading phase. After loading is complete and the application enters a stable running state, its CPU usage drops to a lower and stable level. After you enable CPU Burst, you do not need to configure an excessively high CPU limit for the container. The application can use more time slices during the startup phase, which ensures that the application starts quickly.

Verification steps

1. Use the YAML content of the following example application to create a file named apache-demo.yaml.

   Configure the Annotation in the metadata field of the Pod object to enable the CPU Burst policy for the Pod individually.

   apiVersion: v1
   kind: Pod
   metadata:
     name: apache-demo
     annotations:
       koordinator.sh/cpuBurst: '{"policy": "auto"}'   # Enable the CPU Burst policy.
   spec:
     containers:
     - command:
       - httpd
       - -D
       - FOREGROUND
       image: registry.cn-zhangjiakou.aliyuncs.com/acs/apache-2-4-51-for-slo-test:v0.1
       imagePullPolicy: Always
       name: apache
       resources:
         limits:
           cpu: "4"
           memory: 10Gi
         requests:
           cpu: "4"
           memory: 10Gi
     nodeName: $nodeName # Set to the actual node name.
     hostNetwork: False
     restartPolicy: Never
     schedulerName: default-scheduler

2. Deploy Apache HTTP Server as the target application for evaluation.

   kubectl apply -f apache-demo.yaml

3. Use the wrk2 load testing tool to send requests.

   # Download the wrk2 open source testing tool, and then extract and install it. For more information, see https://github.com/giltene/wrk2.
   # The Gzip compression module is currently enabled in the Apache image configuration to simulate the compute logic that the server uses to process requests.
   # Run the load generation command, and remember to change the IP address to that of the target application.
   ./wrk -H "Accept-Encoding: deflate, gzip" -t 2 -c 12 -d 120 --latency --timeout 2s -R 24 http://$target_ip_address:8010/static/file.1m.test

   Note

   • Change the target address in the command to the IP address of the Apache Pod.

   • Adjust the queries per second (QPS) load on the request side by changing the -R parameter.

Result analysis

The following data shows the performance of Alibaba Cloud Linux and open-source CentOS, before and after the CPU Burst policy is enabled.

• All disabled indicates that the CPU Burst policy is set to none.

• All enabled indicates that the CPU Burst policy is set to auto.

The preceding comparison data shows the following:

• After CPU Burst is enabled, the p99 percentile of the application response time (RT) metric is significantly improved.

• After CPU Burst is enabled, CPU throttling is substantially reduced, while the overall Pod CPU utilization remains basically unchanged.

If I already use CPU Burst through the earlier ack-slo-manager protocol, is it still supported after I upgrade to ack-koordinator?

The earlier Pod protocol requires you to specify alibabacloud.com/cpuBurst in the Annotation. ack-koordinator is fully compatible with this earlier protocol, so you can seamlessly upgrade the add-on to the new version.

ack-koordinatorsupports the protocols in the following versions:

Why does CPU throttling still occur on a Pod after the CPU Burst configuration is enabled?

Common causes and corresponding adjustments are as follows.

• An incorrect configuration format prevents the CPU Burst policy from taking effect. Modify and verify the configuration by following Advanced parameter configuration.

• CPU throttling still occurs when CPU utilization reaches the upper limit configured by cfsQuotaBurstPercent because CPU resources are insufficient.

  Adjust the CPU requests and limits based on your application's actual requirements.

• The CPU Burst policy adjusts two cgroup parameters of a Pod: cpu.cfs_quota_us and cpu.cfs_burst_us. For more information, see Advanced parameter configuration. The cpu.cfs_quota_us parameter is set only after ack-koordinator detects CPU throttling, so there is a slight delay. The cpu.cfs_burst_us parameter is set directly based on the configuration and takes effect more quickly.

  Use Alibaba Cloud Linux for better results.

• The CPU Burst policy has a protection mechanism when it adjusts cpu.cfs_quota_us: the safe node-wide CPU utilization threshold configured by sharePoolThresholdPercent. When overall node utilization is too high, cpu.cfs_quota_us is reset to its initial value to prevent a single Pod from causing further disruption.

  Set the safe node-wide CPU utilization threshold based on the actual requirements of your application to prevent high overall node utilization from affecting application performance.

Must I use Alibaba Cloud Linux to enable the CPU Burst policy?

ack-koordinator supports the CPU Burst policy on all Alibaba Cloud Linux and open-source CentOS kernels. We recommend Alibaba Cloud Linux. With Alibaba Cloud Linux kernel features, ack-koordinator can provide a more fine-grained CPU elasticity management mechanism. For more information, see Enable CPU Burst on the cgroup v1 interface.

Why does the number of threads obtained by application code change after CPU Burst is enabled?

This is because the way CPU Burst works conflicts with the way some applications obtain system resources. ack-koordinator dynamically adjusts the container's underlying cgroup parameter cpu.cfs_quota_us when it implements CPU Burst. This value represents the CPU time quota available to the container in the current scheduling period. ack-koordinator dynamically scales this quota based on the application load.

Many applications (such as Java's Runtime.getRuntime().availableProcessors()) directly read cpu.cfs_quota_us to estimate the number of available CPU cores. Therefore, when the CPU quota is dynamically adjusted, the number of cores obtained by the application also changes. Parameters that depend on this value, such as the thread pool size, become unstable.

Applications should rely on the fixed limits.cpu value defined in the Pod.

1. Inject an environment variable: Use resourceFieldRef to inject the Pod limits.cpu value into the container as an environment variable.

   env: 
     - name: CPU_LIMIT 
       valueFrom: 
         resourceFieldRef: 
           resource: limits.cpu

2. Modify application code: Change the application startup logic so that it reads CPU_LIMIT first when calculating and setting the thread pool size. After configuration, the application runs based on a stable and reliable value regardless of how CPU Burst adjusts the quota.