Container Service for Kubernetes:Enable dynamic resource overcommitment

QoS classes and Batch resources

Kubernetes assigns each pod a quality of service (QoS) class based on its resource configuration. Batch resources are designed specifically for the BestEffort class:

To use dynamic resource overcommitment, set koordinator.sh/qosClass: "BE" on the pod and replace standard resource fields with kubernetes.io/batch-cpu and kubernetes.io/batch-memory.

Step 1: Create the ConfigMap

Create a file named configmap.yaml with the following content:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ack-slo-config
  namespace: kube-system
data:
  # colocation-config controls dynamic Batch resource calculation and updates.
  # Related features: Dynamic resource overcommitment, load-aware scheduling.
  colocation-config: |
    {
      "enable": true,                        # Required: enables Batch resource updates. Setting to false resets reclaimed resources to 0.
      "metricAggregateDurationSeconds": 60,  # How often (seconds) node metrics are aggregated. Use the default value.
      "cpuReclaimThresholdPercent": 60,      # Reclaim threshold for batch-cpu, as a % of allocatable CPU. Default: 65.
      "memoryReclaimThresholdPercent": 70,   # Reclaim threshold for batch-memory, as a % of allocatable memory. Default: 65.
      "memoryCalculatePolicy": "usage"       # How batch-memory capacity is calculated: "usage" (default) or "request".
    }

The cpuReclaimThresholdPercent and memoryReclaimThresholdPercent values in this example (60 and 70) are sample values. The actual defaults are 65 for both parameters.

The following table describes each parameter in detail:

Calculate Batch resource capacity

ack-koordinator applies the following formula to calculate the amount of Batch resources available on each node.

Usage-based calculation (default, memoryCalculatePolicy: "usage"):

nodeBatchAllocatable = nodeAllocatable × thresholdPercent − podUsage(non-BE) − systemUsage

Request-based calculation (memoryCalculatePolicy: "request", applies to batch-memory only):

nodeBatchAllocatable = nodeAllocatable × thresholdPercent − podRequest(non-BE) − systemUsage

Where:

Step 2: Apply the ConfigMap

Check whether the ack-slo-config ConfigMap already exists in the kube-system namespace:

• If it exists, use kubectl patch to merge your changes without overwriting other settings:

  kubectl patch cm -n kube-system ack-slo-config --patch "$(cat configmap.yaml)"

• If it does not exist, create it:

  kubectl apply -f configmap.yaml

Step 1: Check available Batch resources on the node

# Replace $nodeName with the actual node name.
kubectl get node $nodeName -o yaml

Look for the status.allocatable section in the output:

status:
  allocatable:
    # Unit: millicore. The following example shows 50 cores available.
    kubernetes.io/batch-cpu: 50000
    # Unit: bytes. The following example shows 50 GB available.
    kubernetes.io/batch-memory: 53687091200

Step 2: Configure the pod to use Batch resources

Add the koordinator.sh/qosClass: "BE" label to the pod metadata and set kubernetes.io/batch-cpu and kubernetes.io/batch-memory in the container's resources field:

metadata:
  labels:
    # Required: sets the pod's QoS class to BestEffort.
    koordinator.sh/qosClass: "BE"
spec:
  containers:
  - resources:
      requests:
        # Unit: millicore. "1k" = 1000 millicores = 1 core.
        kubernetes.io/batch-cpu: "1k"
        # Unit: bytes.
        kubernetes.io/batch-memory: "1Gi"
      limits:
        kubernetes.io/batch-cpu: "1k"
        kubernetes.io/batch-memory: "1Gi"

After upgrading from ack-slo-manager to ack-koordinator, does the old overcommitment configuration still work?

Yes. ack-koordinator is backward compatible with the earlier ack-slo-manager protocol. The ACK Pro cluster scheduler can calculate requested and available resources using both the old and new protocol formats simultaneously, so you can upgrade without reconfiguring existing workloads.

The earlier protocol uses:

• The alibabacloud.com/qosClass pod annotation

• The alibabacloud.com/reclaimed field for resource requests and limits

ack-koordinator supports these through protocol versions dated no later than July 30, 2023. Migrate existing workloads to the koordinator.sh protocol when convenient.

The following table shows compatibility across component versions:

Why does memory usage spike right after the pod starts?

Symptom: Memory usage jumps immediately after a container starts, exceeding the expected kubernetes.io/batch-memory limit.

Cause: When a container is created, ack-koordinator sets the cgroup memory limit based on kubernetes.io/batch-memory. Some applications read the cgroup limit at startup to determine how much memory to allocate internally. If the application reads the cgroup before ack-koordinator has written the limit, it may allocate more memory than intended. The operating system does not immediately reclaim that memory, so usage stays elevated until it naturally drops below the configured limit.

Check: Run the following command inside the container to confirm the memory limit is set correctly:

# Unit: bytes
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
# Expected output example
1048576000

Fix: Configure the application's memory limit in its startup script before the main process begins. This ensures the limit is in place before the application reads the cgroup.

Why does a BestEffort pod stay in Pending state?

Symptom: A pod configured with Batch resources remains in Pending state and cannot be scheduled.

Check: Run kubectl describe pod <pod-name> and look for scheduling failure events.

Common causes and fixes: