Container Service for Kubernetes:Enable the CPU Burst performance optimization policy

Benefits

CPU utilization is a key metric for measuring the health of a container. Cluster administrators typically use this metric to configure a container's CPU Limit. Compared to the commonly used per-second metrics, a container's CPU utilization at the hundred-millisecond level often shows more pronounced glitches and rapid fluctuations. In the following figure, the CPU usage is significantly less than 4 cores on a per-second basis (purple line). However, on a per-millisecond basis (green line), the container's CPU usage exceeds 4 cores during some periods. If you set the CPU Limit to 4 cores, the operating system suspends threads because of CPU throttling.

The following figures show how CPU resources are allocated to the threads of a web service container on a 4-core node after the container receives a request (req). The container's CPU Limit is 2. The figure on the left shows the normal case. The figure on the right shows the case after CPU Burst is enabled.

In addition to the preceding scenarios, CPU Burst is also suitable for handling CPU usage spikes. For example, when service traffic spikes, ack-koordinator can eliminate CPU bottlenecks within a few seconds while ensuring that the overall load on the node remains at a safe level.

Scenarios

The CPU Burst feature is useful in the following scenarios.

• A container frequently experiences CPU throttling, which affects application performance, even though its CPU usage is usually below the configured CPU Limit. You can enable CPU Burst to let the container use accumulated time slices during load bursts. This effectively resolves CPU throttling issues and improves application service quality.

• A container application consumes more CPU resources during its startup phase. After the application starts and enters a stable running state, its CPU usage drops to a lower, stable level. If you enable CPU Burst, the application can use more time slices during startup to ensure a quick launch without needing to configure an excessively high CPU Limit.

Verification steps

1. Create a file named apache-demo.yaml that contains the following content for the example application.

   Add an annotation to the metadata field of the pod object to enable the CPU Burst policy for the pod.

   apiVersion: v1
   kind: Pod
   metadata:
     name: apache-demo
     annotations:
       koordinator.sh/cpuBurst: '{"policy": "auto"}'   # Enable the CPU Burst policy.
   spec:
     containers:
     - command:
       - httpd
       - -D
       - FOREGROUND
       image: registry.cn-zhangjiakou.aliyuncs.com/acs/apache-2-4-51-for-slo-test:v0.1
       imagePullPolicy: Always
       name: apache
       resources:
         limits:
           cpu: "4"
           memory: 10Gi
         requests:
           cpu: "4"
           memory: 10Gi
     nodeName: $nodeName # Change this to the actual node name.
     hostNetwork: False
     restartPolicy: Never
     schedulerName: default-scheduler

2. Run the following command to deploy Apache HTTP Server as the target application for evaluation.

   kubectl apply -f apache-demo.yaml

3. Use the wrk2 stress testing tool to send requests.

   # Download, decompress, and install the open-source wrk2 testing tool. For details, see https://github.com/giltene/wrk2.
   # The Gzip compression module is enabled in the current Apache image configuration to simulate the server-side request processing logic.
   # Run the stress testing command. Make sure to change the IP address to that of the target application.
   ./wrk -H "Accept-Encoding: deflate, gzip" -t 2 -c 12 -d 120 --latency --timeout 2s -R 24 http://$target_ip_address:8010/static/file.1m.test

   Note

   • Replace the target address in the command with the IP address of the Apache pod.

   • You can modify the -R parameter to adjust the QPS rate from the sender.

Result analysis

The following data shows the performance on Alibaba Cloud Linux and the community version of CentOS before and after the CPU Burst policy is enabled.

• Completely disabled means that the CPU Burst policy is set to none.

• When enabled, the CPU Burst policy is set to auto.

The data comparison shows the following:

• After the CPU Burst feature is enabled, the p99 response time (RT) of the application is significantly improved.

• After the CPU Burst feature is enabled, CPU throttling is significantly reduced, while the overall pod CPU utilization remains almost unchanged.

Does the CPU Burst feature that I enabled based on an earlier version of the ack-slo-manager protocol still work after I upgrade ack-slo-manager to ack-koordinator?

Yes. Earlier versions of the Pod protocol require you to add the alibabacloud.com/cpuBurst annotation. ack-koordinator is fully compatible with these earlier protocol versions. You can seamlessly upgrade the component to the new version.

The following describes the compatibility of ack-koordinator with different protocol versions.

After I enable the CPU Burst configuration, why do my pods still experience CPU throttling?

This may happen for several reasons. Review the following items to troubleshoot the issue.

• The configuration format may be incorrect, which prevents the CPU Burst policy from taking effect. For more information, see Advanced configuration to verify and modify the configuration.

• CPU throttling events may still be generated if CPU utilization reaches the upper limit specified by cfsQuotaBurstPercent due to insufficient CPU resources.

  You can adjust the Request and Limit values based on your application's actual needs.

• The CPU Burst policy adjusts two cgroup parameters for a pod: cpu.cfs_quota_us and cpu.cfs_burst_us. For more information, see Advanced parameter configuration. The cpu.cfs_quota_us parameter is set only after ack-koordinator detects a CPU Throttled event, which causes a slight delay. In contrast, the cpu.cfs_burst_us parameter is set directly based on the configuration, which makes it more responsive.

  For optimal results, we recommend that you use the Alibaba Cloud Linux operating system.

• The CPU Burst policy includes a protection mechanism for adjusting cpu.cfs_quota_us, which is a node-level safety threshold configured by the sharePoolThresholdPercent parameter. If the node's CPU utilization becomes too high, cpu.cfs_quota_us is reset to its initial value to prevent the pod from affecting other pods.

  You can set the safety threshold based on your application's requirements to avoid performance degradation caused by high machine utilization.

Is it necessary to use the Alibaba Cloud Linux operating system to enable the CPU Burst policy?

No. The ack-koordinator CPU Burst policy supports all Alibaba Cloud Linux and CentOS open source kernels. However, we recommend that you use the Alibaba Cloud Linux operating system. The kernel features of Alibaba Cloud Linux allow ack-koordinator to provide a more fine-grained and elastic CPU management mechanism. For more information, see Enable the CPU Burst Feature on the cgroup v1 Interface.

After I enable CPU Burst, why does the number of threads reported by my application's code change?

This occurs because the CPU Burst mechanism can conflict with the way some applications retrieve system resource information. ack-koordinator implements CPU Burst by dynamically adjusting the container's underlying cgroup parameter cpu.cfs_quota_us. This value represents the CPU time quota available to the container within the current scheduling period. ack-koordinator dynamically scales this quota based on the application's load.

Many applications, such as Java's Runtime.getRuntime().availableProcessors(), directly read cpu.cfs_quota_us to infer the number of available CPU cores. Therefore, when the CPU quota is dynamically adjusted, the number of cores retrieved by the application also changes, causing parameters that depend on this value, such as the thread pool size, to become unstable.

We recommend that you modify your application to rely on the fixed limits.cpu value defined in the pod.

1. Inject environment variables: You can use resourceFieldRef to inject the limits.cpu value of the pod into the container as an environment variable.

   env: 
     - name: CPU_LIMIT 
       valueFrom: 
         resourceFieldRef: 
           resource: limits.cpu

2. Modify the application code: You can modify the application startup logic to prioritize reading the CPU_LIMIT environment variable to calculate and set the thread pool size. After this change, the application runs based on a stable and reliable value, regardless of how CPU Burst adjusts the quota.