Container Service for Kubernetes:Work with load-aware hotspot descheduling

Table of contents

• Limits

• Introduction to load-aware hotspot descheduling

  • load-aware pod scheduling

  • How koord-descheduler works

  • Load-aware hotspot descheduling policies

• Step 1: Install or modify ack-koordinator and enable load-aware hotspot descheduling

• Step 2: Enable the LowNodeLoad plug-in

• Step 3: (Optional) Enable the load-aware scheduling plug-in

• Step 4: Verify load-aware hotspot descheduling

• Advanced settings

• FAQ

Limits

• Only Container Service for Kubernetes (ACK) Pro clusters support this feature. For more information about how to create an ACK Pro cluster, see Create an ACK managed cluster.

• To use the load-aware hotspot descheduling feature, make sure that the following requirements are met.

  Component	Required version
  ACK scheduler	v1.22.15-ack-4.0 and later or v1.24.6-ack-4.0 and later
  ack-koordinator(ack-slo-manager)	1.1.1-ack.1 and later
  Helm	3.0 and later

Introduction to load-aware hotspot descheduling

This section describes the terms used in load-aware hotspot descheduling.

Load-aware scheduling

The ACK scheduler supports load-aware scheduling, which can schedule pods to nodes that run with low loads. Due to the changes in the cluster environment, traffic, and requests, the utilization of nodes dynamically changes and may break the load balance between nodes in the cluster, and even result in extreme load imbalance. This affects the runtime quality of the workload. ack-koordinator can identify changes in the loads of nodes and automatically optimize nodes that exceed the safety load to prevent extreme load imbalance. You can use a combination of load-aware scheduling and hotspot descheduling to achieve optimal load balancing among nodes. For more information about load-aware scheduling, see Use load-aware pod scheduling.

How koord-descheduler works

The ack-koordinator component provides the koord-descheduler module. The LowNodeLoad plug-in in koord-descheduler is responsible for sensing the loads on the nodes and reducing load hotspots. Unlike the Kubernetes-native descheduler plug-in LowNodeUtilization, the LowNodeLoad plug-in decides to deschedule nodes based on the actual resource utilization of the nodes, while LowNodeUtilization decides to deschedule based on resource allocation.

Descheduling procedure

koord-descheduler periodically performs descheduling. The following figure shows the steps of descheduling within each cycle.

1. Data collection: collects information about nodes and workloads in the cluster and the resource utilization statistics.

2. Descheduling policy implementation by the policy plug-in.

   This section uses LowNodeLoad as an example.

   1. Identifies hotspot nodes. For more information about the classification of nodes, see Load thresholds.

   2. Traverses all hotspot nodes, identifies the nodes on which pods can be migrated, and sorts the pods. For more information about how pods are sorted, see Pod sorting policy.

   3. Traverses all pods to be migrated and checks whether the pods meet the requirements for migration based on constraints such as the cluster size, resource utilization, and the ratio of replicated pods. For more information, see Load-aware hotspot descheduling policies.

   4. Only pods that meet the requirements are migrated. If no pod meets the requirements on the current node, LowNodeLoad continues to traverse the pods on other hotspot nodes.

3. Pod eviction and migration: evicts the pods that meet the requirements for migration. For more information, see Pod eviction.

Load thresholds

The resource utilization statistics are updated every minute and the average values within the previous 5 minutes are displayed.

1. Idle nodes: nodes whose resource utilization is lower than 45%.

2. Normal nodes: nodes whose resource utilization is higher than or equal to 45% but lower than or equal to 70%. This is the desired resource utilization range for cluster nodes.

3. Hotspot nodes: nodes whose resource utilization is higher than 70. Pods on hotspot nodes will be evicted until the resource utilization of these nodes drops to 70% or lower.

Load-aware hotspot descheduling policies

kubectl get event | grep stress-demo-588f9646cf-7****
55s         Normal    Evicting           podmigrationjob/3bf8f623-4d10-4fc5-ab4e-2bead3c4****   Pod "default/stress-demo-588f9646cf-7****" evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(76.72%)>threshold(50.00%)"
22s         Normal    EvictComplete      podmigrationjob/3bf8f623-4d10-4fc5-ab4e-2bead3c4****   Pod "default/stress-demo-588f9646cf-7****" has been evicted
55s         Normal    Descheduled        pod/stress-demo-588f9646cf-7****                       Pod evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(76.72%)>threshold(50.00%)"
55s         Normal    Killing            pod/stress-demo-588f9646cf-7****                       Stopping container stress

Step 1: Install or modify ack-koordinator and enable load-aware hotspot descheduling

Step 2: Enable the LowNodeLoad plug-in

1. Create a file named koord-descheduler-config.yaml and add the following content to the file.

   koord-descheduler-config.yaml is a ConfigMap object used to enable the LowNodeLoad plug-in.

   Click to view details

   # koord-descheduler-config.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: koord-descheduler-config
     namespace: kube-system
   data:
     koord-descheduler-config: |
       # Do not modify the following system configuration of koord-desheduler. 
       apiVersion: descheduler/v1alpha2
       kind: DeschedulerConfiguration
       leaderElection:
         resourceLock: leases
         resourceName: koord-descheduler
         resourceNamespace: kube-system
       deschedulingInterval: 120s # The interval at which LowNodeLoad runs. The interval is set to 120 seconds in this example. 
       dryRun: false # The global read-only mode. After you enable this mode, koord-descheduler does not perform any operations. 
       # The preceding configuration is the system configuration. 
   
       profiles:
       - name: koord-descheduler
         plugins:
           deschedule:
             disabled:
               - name: "*"
           balance:
             enabled:
               - name: LowNodeLoad # Enable the LowNodeLoad plug-in. 
           evict:
             disabled:
               - name: "*"
             enabled:
               - name: MigrationController # Enable the migration controller. 
   
         pluginConfig:
         - name: MigrationController # Configure the parameters of the migration controller. 
           args:
             apiVersion: descheduler/v1alpha2
             kind: MigrationControllerArgs
             defaultJobMode: EvictDirectly
   
         - name: LowNodeLoad # Configure the LowNodeLoad plug-in. 
           args:
             apiVersion: descheduler/v1alpha2
             kind: LowNodeLoadArgs
   
             lowThresholds:  # Specify the low resource threshold for identifying idle nodes. Nodes whose load is lower than the threshold are considered idle nodes. 
               cpu: 20 # The low CPU utilization threshold is 20%. 
               memory: 30  # The low memory utilization threshold is 30%. 
             highThresholds: # Specify the high resource threshold for identifying hotspot nodes. Nodes whose load is higher than the threshold are considered hotspot nodes. 
               cpu: 50  # The high CPU utilization threshold is 50%. 
               memory: 60 # The high memory utilization threshold is 60%. 
   
             evictableNamespaces: # Specify the namespaces that you want to include or exclude for descheduling. The include and exclude lists are mutually exclusive. 
               include: # Specify the namespaces that you want to include for descheduling. 
                 - default
               # exclude: # Specify the namespaces that you want to exclude for descheduling. 
                 # - "kube-system"
                 # - "koordinator-system"

2. Run the following command to apply the configuration to the cluster:

   kubectl apply -f koord-descheduler-config.yaml

3. Run the following command to restart koord-descheduler.

   After koord-descheduler is restarted, the modified configuration takes effect.

   kubectl -n kube-system scale deploy ack-koord-descheduler --replicas 0
   deployment.apps/ack-koord-descheduler scaled
   kubectl -n kube-system scale deploy ack-koord-descheduler --replicas 1
   deployment.apps/ack-koord-descheduler scaled

Step 3: (Optional) Enable the load-aware scheduling plug-in

For more information about how to enable the load-aware scheduling plug-in to achieve optimal load balancing among nodes, see Step 1 in the Use load-aware pod scheduling topic.

Step 4: Verify load-aware hotspot descheduling

In this section, a cluster that contains three nodes is used as an example. Each node has 104 vCores and 396 GB of memory.

1. Create a file named stress-demo.yaml and add the following content to the file.

   Click to view details

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: stress-demo
     namespace: default
     labels:
       app: stress-demo
   spec:
     replicas: 6
     selector:
       matchLabels:
         app: stress-demo
     template:
       metadata:
         name: stress-demo
         labels:
           app: stress-demo
       spec:
         containers:
           - args:
               - '--vm'
               - '2'
               - '--vm-bytes'
               - '1600M'
               - '-c'
               - '2'
               - '--vm-hang'
               - '2'
             command:
               - stress
             image: polinux/stress
             imagePullPolicy: Always
             name: stress
             resources:
               limits:
                 cpu: '2'
                 memory: 4Gi
               requests:
                 cpu: '2'
                 memory: 4Gi
         restartPolicy: Always

2. Run the following command to create a pod for stress testing:

   kubectl create -f stress-demo.yaml
   deployment.apps/stress-demo created

3. Run the following command to view the status of the pod until it starts to run.

   kubectl get pod -o wide

   Expected output:

   NAME                           READY   STATUS    RESTARTS   AGE   IP            NODE                    NOMINATED NODE   READINESS GATES
   stress-demo-588f9646cf-s****   1/1     Running   0          82s   10.XX.XX.53   cn-beijing.10.XX.XX.53   <none>           <none>

   The output indicates that pod stress-demo-588f9646cf-s**** is scheduled to node cn-beijing.10.XX.XX.53.

4. Increase the load of node cn-beijing.10.XX.XX.53. Then, run the following command to check the load of each node:

   kubectl top node

   Expected output:

   NAME                      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
   cn-beijing.10.XX.XX.215   17611m       17%    24358Mi         6%
   cn-beijing.10.XX.XX.53    63472m       63%    11969Mi         3%

   The output indicates that the load of node cn-beijing.10.XX.XX.53 is higher, which is 63%. The load exceeds the high resource threshold of 50%. The load of node cn-beijing.10.XX.XX.215 is lower, which is 17%. Th load is lower than the low resource threshold of 20%.

5. Enable load-aware hotspot descheduling. For more information, see Step 2: Enable the LowNodeLoad plug-in.

6. Run the following command to view the changes of the pods.

   Wait for the descheduler to identify hotspot nodes and evict pods.

   Note

   A node is considered a hotspot node if the node consecutively exceeds the high resource threshold five times within 10 minutes.

   kubectl get pod -w

   Expected output:

   NAME                           READY   STATUS               RESTARTS   AGE     IP           NODE                     NOMINATED NODE   READINESS GATES
   stress-demo-588f9646cf-s****   1/1     Terminating          0          59s   10.XX.XX.53    cn-beijing.10.XX.XX.53     <none>           <none>
   stress-demo-588f9646cf-7****   1/1     ContainerCreating    0          10s   10.XX.XX.215   cn-beijing.10.XX.XX.215    <none>           <none>

7. Run the following command to view the event:

   kubectl get event | grep stress-demo-588f9646cf-s****

   Expected output:

   2m14s       Normal    Evicting            podmigrationjob/00fe88bd-8d4c-428d-b2a8-d15bcdeb****   Pod "default/stress-demo-588f9646cf-s****" evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(68.53%)>threshold(50.00%)"
   101s        Normal    EvictComplete       podmigrationjob/00fe88bd-8d4c-428d-b2a8-d15bcdeb****   Pod "default/stress-demo-588f9646cf-s****" has been evicted
   2m14s       Normal    Descheduled         pod/stress-demo-588f9646cf-s****                       Pod evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(68.53%)>threshold(50.00%)"
   2m14s       Normal    Killing             pod/stress-demo-588f9646cf-s****                       Stopping container stress

   The output indicates the migration result. The pods on the hotspot node are migrated to the idle node.

Advanced settings

Advanced settings for koord-descheduler

The configuration of koord-descheduler is stored in a ConfigMap. The following code block shows the advanced settings for load-aware hotspot descheduling.

# koord-descheduler-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: koord-descheduler-config
  namespace: kube-system
data:
  koord-descheduler-config: |
    # Do not modify the following system configuration of koord-desheduler. 
    apiVersion: descheduler/v1alpha2
    kind: DeschedulerConfiguration
    leaderElection:
      resourceLock: leases
      resourceName: koord-descheduler
      resourceNamespace: kube-system
    deschedulingInterval: 120s # The interval at which LowNodeLoad runs. The interval is set to 120 seconds in this example. 
    dryRun: false # The global read-only mode. After you enable this mode, koord-descheduler does not perform any operations. 
    # The preceding configuration is the system configuration. 

    profiles:
    - name: koord-descheduler
      plugins:
        deschedule:
          disabled:
            - name: "*"
        balance:
          enabled:
            - name: LowNodeLoad # Enable the LowNodeLoad plug-in. 
        evict:
          disabled:
            - name: "*"
          enabled:
            - name: MigrationController # Enable the migration controller. 

      pluginConfig:
      - name: MigrationController # Configure the parameters of the migration controller. 
        args:
          apiVersion: descheduler/v1alpha2
          kind: MigrationControllerArgs
          defaultJobMode: EvictDirectly
          maxMigratingPerNode: 1 # The maximum number of pods that can be migrated at the same time on a node. 
          maxMigratingPerNamespace: 1  # The maximum number of pods that can be migrated at the same time in a namespace. 
          maxMigratingPerWorkload: 1 # The maximum number of pods that can be migrated at the same time in a workload, such as a Deployment. 
          maxUnavailablePerWorkload: 2 # The maximum number of unavailable replicated pods that are allowed in a workload, such as a Deployment. 
          evictLocalStoragePods: false # Specify whether pods that are configured with the emptyDir or hostPath can be descheduled.
          objectLimiters:
            workload: # Control workload-specific pod migration. By default, the system can migrate only one replicated pod within 5 minutes after the first eviction. 
              duration: 5m
              maxMigrating: 1

      - name: LowNodeLoad # Configure the LowNodeLoad plug-in. 
        args:
          apiVersion: descheduler/v1alpha2
          kind: LowNodeLoadArgs

          lowThresholds:  # Specify the low resource threshold for identifying idle nodes. Nodes whose load is lower than the threshold are considered idle nodes. 
            cpu: 20 # The low CPU utilization threshold is 20%. 
            memory: 30  # The low memory utilization threshold is 30%. 
          highThresholds: # Specify the high resource threshold for identifying hotspot nodes. Nodes whose load is higher than the threshold are considered hotspot nodes. 
            cpu: 50  # The high CPU utilization threshold is 50%. 
            memory: 60 # The high memory utilization threshold is 60%. 

          anomalyCondition: # The hotspot node detection setting. 
            consecutiveAbnormalities: 5 # A node is considered a hotspot node if the node exceeds the highThresholds within five consecutive hotspot detection cycles. Hotspot nodes are descheduled and then the counter is reset. 

          evictableNamespaces: # Specify the namespaces that you want to include or exclude for descheduling. The include and exclude lists are mutually exclusive. 
            include: # Specify the namespaces that you want to include for descheduling. 
              - default
            # exclude: # Specify the namespaces that you want to exclude for descheduling. 
              # - "kube-system"
              # - "koordinator-system"

          nodeSelector: # Only the specified nodes can be descheduled. 
            matchLabels:
              alibabacloud.com/nodepool-id: np77f520e1108f47559e63809713ce****

          podSelectors: # Only the specified pods can be descheduled. 
          - name: lsPods
            selector:
              matchLabels:
                koordinator.sh/qosClass: "LS"

koord-descheduler system settings

Migration control settings

type MigrationObjectLimiter struct {
    Duration time.Duration `json:"duration,omitempty"`
    MaxMigrating *intstr.IntOrString `json:"maxMigrating,omitempty"`
}

objectLimiters:
  workload:
    duration: 5m
    maxMigrating: 1

LowNodeLoad settings

highThresholds:
  cpu: 55 # The high CPU utilization threshold is set to 55%. 
  memory: 75 # The high memory utilization threshold is set to 75%. 

lowThresholds:
  cpu: 25 # The low CPU utilization threshold is set to 25%. 
  memory: 25 # The low memory utilization threshold is set to 25%. 

evictableNamespaces:
  exclude:
    - "kube-system"
    - "koordinator-system"

type PodSelector struct {
    name     string
    selector metav1.LabelSelector
}

# Only Latency-Sensitive (LS) pods can be descheduled. 
podSelectors:
- name: lsPods
  selector:
    matchLabels:
      koordinator.sh/qosClass: "LS"

FAQ

Issue 1: What do I do if the resource utilization of a node has reached the high threshold but no pod on the node is evicted?

The following table describes the possible causes.

Issue 2: Why does the descheduler frequently restart?

The format of the ConfigMap of the descheduler is invalid or the ConfigMap does not exist. Refer to Advanced settings and check the content and format of the ConfigMap, modify the ConfigMap, and then restart the descheduler. For more information about how to restart the descheduler, see Step 2: Enable the LowNodeLoad plug-in.

Issue 3: What do I do if new pods are scheduled to the node on which pods are evicted?

After the system evicts the pods on a node, the system automatically recreates and schedules the pods. The scheduler selects nodes based on the specified configuration. We recommend that you enable load-aware pod scheduling for the scheduler and use this feature together with descheduling. This issue is more likely to occur if you have specified the nodes to which pods can be scheduled, only a few number of idle nodes are available, and the resource utilizations of these nodes are close. In this scenario, you can disable the nodeSelector parameter.