Technical architecture, hybrid resource model, and node-level QoS of colocation - Container Service for Kubernetes

This topic describes the technical architecture, hybrid resource model, and node-level QoS for colocating online and offline workloads, to help you understand and use colocation in ACK.

Background

Colocation deploys different workload types in the same cluster and on the same node to improve resource utilization. Workloads are classified by Service Level Objectives (SLOs): latency-sensitive (LS) workloads typically have QPS or response time (RT) targets and receive high-priority QoS, while best-effort (BE) workloads are compute-intensive, fault-tolerant, and receive low-priority QoS.

Different roles focus on different aspects of colocation:

Cluster resource administrator: monitors resource limits, allocation, and usage per workload to improve utilization and reduce costs.
LS workload administrator: mitigates container interference from resource contention, which can cause elevated 90th/99th percentile latency and degrade service quality.
BE workload administrator: uses classified resource overcommitment to meet workload SLOs.

ACK provides the following colocation mechanisms:

Provides a colocation QoS model with configurable resource priorities.
Provides stable and reliable resource overcommitment.
Supports fine-grained orchestration and isolation of Kubernetes resources.
Provides enhanced workload scheduling.

Architecture

ACK uses ack-koordinator to meet workload SLOs in colocation scenarios. ack-koordinator consists of an SLO controller (a Kubernetes extension, deployed as a Deployment) and an SLO agent (deployed as a DaemonSet) that extends the kubelet with colocation capabilities.

The SLO-aware colocation solution uses CRDs to record node metrics, QoS configurations, and policy enforcement, and tracks resources available for dynamic overcommitment as standard extended resources. Each component provides the following features:

SLO controller: monitors node loads, overcommits resources, and guarantees SLOs based on resource profiles.
Recommender: profiles resources and estimates peak workload demand to simplify container resource configuration.
Koordlet: monitors node loads, detects anomalies, and dynamically isolates resources to suppress interference in a closed loop.
ACK scheduler: optimizes SLO-aware colocation, such as spreading pods during dynamic overcommitment.
Koordinator Descheduler: deployed as a Deployment for pod rescheduling.

Resource model

Kubernetes manages container resources through requests and limits. To prevent contention, administrators often over-provision LS workloads, leaving requested resources underutilized.

resource over-provisioning

The green block shows resources available for dynamic overcommitment. These resources are allocated to BE workloads to meet SLOs and improve overall utilization.

ack-koordinator quantifies overcommittable resources, calculates reclaimed resources in real time, and synchronizes them as standard extended resources to Kubernetes node metadata.

Example YAML template of the node:

status:
  allocatable:
    # milli-core
    kubernetes.io/batch-cpu: 50000
    # bytes
    kubernetes.io/batch-memory: 50000
  capacity:
    kubernetes.io/batch-cpu: 50000
    kubernetes.io/batch-memory: 100000

BE pods have lower priority than LS pods. To use reclaimed resources, add the qos and batch fields to the BE pod YAML. qos: LS sets high priority; qos: BE sets low priority. batch-cpu and batch-memory specify the pod's resource requests. See Enable dynamic resource overcommitment.

Example YAML template of the BE pod:

metadata:
  labels:
    koordinator.sh/qosClass: "BE" # Set the QoS class to BE or LS. 
spec:
  containers:
  - resources:
      limits:
        kubernetes.io/batch-cpu: 1000
        kubernetes.io/batch-memory: 2048
      requests:
        kubernetes.io/batch-cpu: 1000
        kubernetes.io/batch-memory: 2048

Node-level QoS

CPU QoS

CPU QoS, based on Alibaba Cloud Linux, reserves CPU resources for LS pods by configuring Linux scheduling priorities through ack-koordinator's group identity feature. In colocation environments, LS pods receive high priority and BE pods receive low priority, preventing resource contention and ensuring LS service quality.

Benefits of CPU QoS:

Minimized task wake-up latency for LS workloads.
BE task wake-ups do not affect LS pod performance.
BE tasks cannot use the SMT scheduler to share CPU cores, further reducing impact on LS pod performance.

CPU Suppress

The amount of dynamically overcommittable resources varies with LS pod usage and can be allocated to BE pods. CPU Suppress limits BE pod CPU usage to ensure LS pods on the node have sufficient resources.

In the figure, CPU Threshold is the node's CPU usage threshold, Pod (LS).Usage is LS pod CPU usage, and CPU Restriction for BE is BE pod CPU usage. BE pod CPU allocation adjusts with LS pod usage fluctuations, allowing BE pods to exploit idle resources while preventing contention when LS loads increase.

CPU Burst

Kubernetes CPU limits cap container CPU usage per time period. For example, CPU Limit=2 limits a container to 200 ms of CPU time per 100 ms period.

The figure shows thread allocation for a web application container on a four-vCore node with CPU limit set to 2. Despite low overall CPU utilization, Thread 2 cannot resume until the third 100 ms period because CPU throttling occurs in the second period. This increases response time (RT) and causes long-tail latency.

long-tail RT latency

CPU Burst resolves long-tail latency in LS workloads by allowing containers to accumulate idle CPU time slices for handling demand spikes. ACK supports CPU Burst on all compatible kernel versions. For incompatible kernels, ACK monitors CPU throttling and dynamically adjusts container CPU limits to achieve a similar effect.

CPU Burst

Memory QoS

Containers are subject to the following memory limits:

Container memory limit: when memory usage (including page cache) approaches the limit, the OS kernel triggers memory reclaim, which may prevent the application from requesting or releasing memory as expected.
Node memory limit: when a container's memory limit exceeds its request, it can overcommit memory. If available node memory becomes insufficient, the OS kernel reclaims memory from containers, which can severely degrade application performance in colocation scenarios.

ack-koordinator works with Alibaba Cloud Linux to enable memory QoS for pods. It automatically configures the memcg based on container settings and enables the memcg QoS, backend asynchronous reclaim, and global minimum watermark rating features. This optimizes memory-sensitive application performance while ensuring fair memory scheduling.

Memory QoS features:

When pod memory usage approaches its limit, the memcg asynchronously reclaims memory to avoid synchronous full reclaim, minimizing impact on application performance.
When node memory is insufficient, memory is reclaimed preferentially from pods whose usage exceeds their request, preventing overcommitting pods from degrading others.
LS pod memory requests are prioritized, reducing the likelihood of triggering full node memory reclaim.
- memory.limit_in_bytes: the upper limit of memory that can be used by a pod.
- memory.high: the memory throttling threshold.
- memory.wmark_high: the memory reclaim threshold.
- memory.min: the memory lock threshold.

Resource isolation based on the L3 cache and MBA

Colocated containers share the node's L3 cache (last-level cache), with memory bandwidth controlled by Memory Bandwidth Allocation (MBA). ECS Bare Metal Instance (EBM) provides the LLC feature to dynamically adjust pod CPU cache and the MBA feature to control memory bandwidth distribution. ack-koordinator further limits BE pod resources in a fine-grained manner to protect LS pod performance.

Next steps

See Getting started to build an LS and BE colocation environment with ack-koordinator. Related colocation features: