Container Service for Kubernetes:Use hotspot-aware descheduling to balance node loads

Limits

• Only ACK managed Pro clusters are supported.

• The following component versions are required:

  Component	Version
  ACK Scheduler	v1.22.15-ack-4.0 or later, v1.24.6-ack-4.0 or later
  ack-koordinator	v1.1.1-ack.1 or later
  Helm	v3.0 or later

Billing

ack-koordinator is free to install and use. Additional charges may apply in these scenarios:

• Worker node resources: ack-koordinator is a self-managed component and consumes worker node resources after installation. Configure resource requests for each module when you install the component.

• Prometheus custom metrics: If you enable Enable Prometheus Monitoring for ACK-Koordinator and use the Alibaba Cloud Prometheus service, the exposed metrics are counted as custom metrics and incur fees. Fees depend on cluster size and number of applications. Review the billing of Prometheus instances documentation before enabling this feature. Query usage data to monitor and manage your resource consumption.

How it works

Execution cycle

The Koordinator Descheduler runs periodically. Each execution cycle has three stages:

1. Data collection: Fetches node and workload information along with their resource utilization data.

2. Plugin execution (using LowNodeLoad as the example):

   1. Identifies hot spot nodes based on highThresholds and lowThresholds.

   2. Traverses all hot spot nodes, scores the eligible Pods, and sorts them for eviction. See Pod scoring policy.

   3. Checks each candidate Pod against migration constraints—cluster capacity, resource utilization, and replica ratio limits. See Load-aware hot spot descheduling policies.

   4. Marks Pods that pass all checks as migration candidates and skips those that do not.

3. Pod eviction and migration: Evicts the migration candidates using the eviction API.

Node classification

The LowNodeLoad plugin classifies nodes into three categories based on two thresholds: lowThresholds (idle threshold) and highThresholds (hot spot threshold).

Using lowThresholds = 45% and highThresholds = 70% as an example:

1. Idle node: Resource utilization is below lowThresholds (< 45%).

2. Normal node: Resource utilization is between lowThresholds and highThresholds (45%–70%). This is the target range.

3. Hot spot node: Resource utilization exceeds highThresholds (> 70%). Pods are evicted until the node load drops to 70% or below.

Resource utilization data is updated every minute and reflects the 5-minute average.

If all nodes in the cluster are above lowThresholds, the overall cluster load is considered high. In this state, the Koordinator Descheduler suspends descheduling even if some nodes exceed highThresholds.

Prerequisites

Before you begin, make sure you have:

• An ACK managed Pro cluster

• ACK Scheduler v1.22.15-ack-4.0 or later, ack-koordinator v1.1.1-ack.1 or later, and Helm v3.0 or later

Step 1: Enable descheduling in ack-koordinator

• New installation: Install ack-koordinator and select Enable Descheduling For Ack-koordinator on the component configuration page. See Install the ack-koordinator component.

• Existing installation: On the component configuration page, select Enable Descheduling For Ack-koordinator. See Modify the ack-koordinator component.

Step 2: Enable the LowNodeLoad plugin

1. Create a koord-descheduler-config.yaml file with the following content. This ConfigMap enables the LowNodeLoad descheduling plugin.

   # koord-descheduler-config.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: koord-descheduler-config
     namespace: kube-system
   data:
     koord-descheduler-config: |
       # System configuration for koord-descheduler. Do not modify this section.
       apiVersion: descheduler/v1alpha2
       kind: DeschedulerConfiguration
       leaderElection:
         resourceLock: leases
         resourceName: koord-descheduler
         resourceNamespace: kube-system
       deschedulingInterval: 120s  # Execution interval. The descheduler runs every 120s.
                                   # Must not exceed detectorCacheTimeout (default: 5m).
       dryRun: false               # Set to true to run in read-only mode (no evictions).
       # End of system configuration.
   
       profiles:
       - name: koord-descheduler
         plugins:
           balance:
             enabled:
               - name: LowNodeLoad      # Enable the LowNodeLoad plugin for hot spot descheduling.
           evict:
             enabled:
               - name: MigrationController  # Enable the eviction and migration controller.
   
         pluginConfig:
         - name: MigrationController
           args:
             apiVersion: descheduler/v1alpha2
             kind: MigrationControllerArgs
             defaultJobMode: EvictDirectly
   
         - name: LowNodeLoad
           args:
             apiVersion: descheduler/v1alpha2
             kind: LowNodeLoadArgs
   
             # A node is idle if usage of ALL resources is below lowThresholds.
             lowThresholds:
               cpu: 20    # 20% CPU utilization
               memory: 30 # 30% memory utilization
   
             # A node is a hot spot if usage of ANY resource exceeds highThresholds.
             highThresholds:
               cpu: 50    # 50% CPU utilization
               memory: 60 # 60% memory utilization
   
             # Scopes descheduling to specific namespaces.
             # include and exclude are mutually exclusive—configure only one.
             evictableNamespaces:
               include:
                 - default
               # exclude:
               #   - "kube-system"
               #   - "koordinator-system"

2. Apply the ConfigMap to the cluster.

   kubectl apply -f koord-descheduler-config.yaml

3. Restart the Koordinator Descheduler to load the new configuration.

   kubectl -n kube-system scale deploy ack-koord-descheduler --replicas 0
   kubectl -n kube-system scale deploy ack-koord-descheduler --replicas 1

Step 3 (optional): Enable load-aware scheduling

For optimal load balancing, enable load-aware scheduling so the ACK Scheduler avoids placing Pods on hot spot nodes after eviction. See Step 1: Enable load-aware scheduling.

Set loadAwareThreshold in the load-aware scheduling policy to the same value as highThresholds. If the values differ, evicted Pods may be rescheduled back onto hot spot nodes—especially when only a few nodes are available with similar utilization levels.

Step 4: Verify descheduling

This example uses a three-node cluster where each node has 104 cores and 396 GiB of memory.

1. Create a stress-demo.yaml file with the following content.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: stress-demo
     namespace: default
     labels:
       app: stress-demo
   spec:
     replicas: 6
     selector:
       matchLabels:
         app: stress-demo
     template:
       metadata:
         name: stress-demo
         labels:
           app: stress-demo
       spec:
         containers:
           - args:
               - '--vm'
               - '2'
               - '--vm-bytes'
               - '1600M'
               - '-c'
               - '2'
               - '--vm-hang'
               - '2'
             command:
               - stress
             image: polinux/stress
             imagePullPolicy: Always
             name: stress
             resources:
               limits:
                 cpu: '2'
                 memory: 4Gi
               requests:
                 cpu: '2'
                 memory: 4Gi
         restartPolicy: Always

2. Deploy the stress-testing workload.

   kubectl create -f stress-demo.yaml

   Expected output:

   deployment.apps/stress-demo created

3. Verify the Pods are running and note which node they are scheduled to.

   kubectl get pod -o wide

   Expected output:

   NAME                           READY   STATUS    RESTARTS   AGE   IP            NODE                     NOMINATED NODE   READINESS GATES
   stress-demo-588f9646cf-s****   1/1     Running   0          82s   10.XX.XX.53   cn-beijing.10.XX.XX.53   <none>           <none>

4. Increase the load on cn-beijing.10.XX.XX.53 and check node utilization.

   kubectl top node

   Expected output:

   NAME                      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
   cn-beijing.10.XX.XX.215   17611m       17%    24358Mi         6%
   cn-beijing.10.XX.XX.53    63472m       63%    11969Mi         3%

   Node cn-beijing.10.XX.XX.53 is at 63% CPU, which exceeds the configured hot spot threshold of 50%. Node cn-beijing.10.XX.XX.215 is at 17% CPU, which is below the idle threshold of 20%.

5. Enable the LowNodeLoad plugin as described in Step 2: Enable the LowNodeLoad plugin.

6. Watch for Pod changes.

   By default, a node must exceed highThresholds for five consecutive checks before it is classified as a hot spot. With the default 120-second interval, this takes approximately 10 minutes.

   kubectl get pod -w

   Expected output:

   NAME                           READY   STATUS             RESTARTS   AGE   IP             NODE                     NOMINATED NODE   READINESS GATES
   stress-demo-588f9646cf-s****   1/1     Terminating        0          59s   10.XX.XX.53    cn-beijing.10.XX.XX.53   <none>           <none>
   stress-demo-588f9646cf-7****   1/1     ContainerCreating  0          10s   10.XX.XX.215   cn-beijing.10.XX.XX.215  <none>           <none>

7. Check the eviction event.

   kubectl get event | grep stress-demo-588f9646cf-s****

   Expected output:

   2m14s   Normal   Evicting        podmigrationjob/00fe88bd-****   Pod "default/stress-demo-588f9646cf-s****" evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(68.53%)>threshold(50.00%)"
   101s    Normal   EvictComplete   podmigrationjob/00fe88bd-****   Pod "default/stress-demo-588f9646cf-s****" has been evicted
   2m14s   Normal   Descheduled     pod/stress-demo-588f9646cf-s****   Pod evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(68.53%)>threshold(50.00%)"
   2m14s   Normal   Killing         pod/stress-demo-588f9646cf-s****   Stopping container stress

   The Pod on the hot spot node has been evicted and migrated to cn-beijing.10.XX.XX.215.

Advanced configuration

All Koordinator Descheduler parameters are set in the ConfigMap created in Step 2. The following shows the full advanced configuration for load-aware hot spot descheduling.

# koord-descheduler-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: koord-descheduler-config
  namespace: kube-system
data:
  koord-descheduler-config: |
    # System configuration. Do not modify this section.
    apiVersion: descheduler/v1alpha2
    kind: DeschedulerConfiguration
    leaderElection:
      resourceLock: leases
      resourceName: koord-descheduler
      resourceNamespace: kube-system
    deschedulingInterval: 120s  # Execution interval. Must not exceed detectorCacheTimeout.
    dryRun: false               # Set to true for read-only mode (no evictions).
    # End of system configuration.

    profiles:
    - name: koord-descheduler
      plugins:
        deschedule:
          disabled:
            - name: "*"  # All plugins disabled by default (shown for reference only).
        balance:
          enabled:
            - name: LowNodeLoad
        evict:
          disabled:
            - name: "*"  # All plugins disabled by default (shown for reference only).
          enabled:
            - name: MigrationController

      pluginConfig:
      - name: MigrationController
        args:
          apiVersion: descheduler/v1alpha2
          kind: MigrationControllerArgs
          defaultJobMode: EvictDirectly
          maxMigratingPerNode: 1       # Max Pods migrating simultaneously per node. 0 = unlimited.
          maxMigratingPerNamespace: 1  # Max Pods migrating simultaneously per namespace.
          maxMigratingPerWorkload: 1   # Max Pods migrating simultaneously per workload (e.g., Deployment).
          maxUnavailablePerWorkload: 2 # Max unavailable replicas per workload during migration.
          evictLocalStoragePods: false # Whether to evict Pods using HostPath or emptyDir volumes.
          objectLimiters:
            workload:               # At most 1 replica migrated per workload within 5 minutes.
              duration: 5m
              maxMigrating: 1

      - name: LowNodeLoad
        args:
          apiVersion: descheduler/v1alpha2
          kind: LowNodeLoadArgs

          lowThresholds:
            cpu: 20
            memory: 30
          highThresholds:
            cpu: 50
            memory: 60

          anomalyCondition:
            consecutiveAbnormalities: 5  # Number of consecutive checks above highThresholds
                                          # before a node is classified as a hot spot.
                                          # Counter resets after eviction.

          detectorCacheTimeout: "5m"  # Cache duration for hot spot checks.
                                      # Must be >= deschedulingInterval.

          evictableNamespaces:
            include:
              - default
            # exclude:
            #   - "kube-system"
            #   - "koordinator-system"

          nodeSelector:  # Process only the specified nodes.
            matchLabels:
              alibabacloud.com/nodepool-id: np77f520e1108f47559e63809713ce****

          podSelectors:  # Process only the specified Pods.
          - name: lsPods
            selector:
              matchLabels:
                koordinator.sh/qosClass: "LS"

FAQ

What's next

• Use load-aware pod scheduling

• Install and manage the ack-koordinator component