Container Service for Kubernetes:Use load-based hot spot descheduling

Limits

• Only ACK managed Pro clusters are supported.

• The related components must meet the following version requirements.

  Component	Version requirements
  ACK Scheduler	v1.22.15-ack-4.0 or later, v1.24.6-ack-4.0 or later
  ack-koordinator	v1.1.1-ack.1 or later
  Helm	v3.0 or later

Billing

The ack-koordinator component is free to install and use. However, additional fees may be incurred in the following scenarios:

• ack-koordinator is a self-managed component and consumes worker node resources after installation. You can configure the resource requests for each module when you install the component.

• By default, ack-koordinator exposes monitoring metrics for features such as resource profiling and fine-grained scheduling in Prometheus format. If you select the Enable Prometheus Monitoring for ACK-Koordinator option when you configure the component and use the Alibaba Cloud Prometheus service, these metrics are considered custom metrics and incur fees. The fees depend on factors such as your cluster size and the number of applications. Before you enable this feature, carefully read the Billing of Prometheus instances documentation for Alibaba Cloud Prometheus to understand the free quota and billing policies for custom metrics. You can monitor and manage your resource usage by querying usage data.

Introduction to load hotspot descheduling

Load-aware scheduling

The ACK scheduler supports load-aware scheduling, which can schedule pods to nodes that run with low loads. Because the cluster environment, traffic, and requests change, node utilization also changes dynamically. This can disrupt the load balance between nodes in the cluster and even result in severe load imbalances. This affects the runtime quality of the workload. ack-koordinator can identify changes in node loads and automatically deschedule pods from nodes that exceed a safe load threshold to prevent severe load imbalances. You can combine load-aware scheduling with hot spot descheduling to achieve optimal load balancing among nodes. For more information, see Use load-aware pod scheduling.

How the Koordinator Descheduler module works

The ack-koordinator component provides the Koordinator Descheduler module. In this module, the LowNodeLoad plug-in detects load levels and performs load-based hot spot descheduling. Unlike the LowNodeUtilization plug-in of the native Kubernetes descheduler, the LowNodeLoad plug-in makes descheduling decisions based on the actual resource utilization of nodes, whereas the LowNodeUtilization plug-in makes decisions based on the resource allocation rate.

Execution procedure

The Koordinator Descheduler module runs periodically. Each execution cycle consists of the following three stages.

1. Data collection: Obtains information about the nodes and workloads in the cluster and their related resource utilization data.

2. Policy plug-in execution.

   The following steps use the LowNodeLoad plug-in as an example.

   1. Identifies hot spot nodes. For more information about the classification of nodes, see LowNodeLoad load threshold parameters.

   2. Traverses all hot spot nodes, identifies the pods that can be migrated, and sorts the pods. For more information about how pods are scored and sorted, see Pod scoring policy.

   3. Traverses all pods to be migrated and checks whether the pods meet the requirements for migration based on constraints such as the cluster size, resource utilization, and the ratio of replicated pods. For more information, see Load-aware hot spot descheduling policies.

   4. If a pod meets the conditions, it is classified as a replica to be migrated. If not, the process continues to traverse other pods and hot spot nodes.

3. Pod eviction and migration: Evicts the pods that meet the requirements for migration. For more information, see API-initiated Eviction.

LowNodeLoad Load Threshold Parameters

For example, in the following figure, lowThresholds is set to 45% and highThresholds is set to 70%. The nodes are classified based on the following criteria. Similarly, if the values of lowThresholds and highThresholds change, the node classification criteria also change accordingly.

By default, resource utilization data is updated every minute. The granularity is the average value over the last 5 minutes.

• Idle Node: A node with resource utilization below 45%.

• Normal Node: A node with resource utilization greater than or equal to 45% and less than or equal to 70%. This load level is the desired range.

• Hot Spot Node: A node with resource utilization above 70%. Some pods on a hot spot node are evicted to lower its load level to 70% or less.

Load hotspot descheduling policies

kubectl get event | grep stress-demo-588f9646cf-7****
55s         Normal    Evicting           podmigrationjob/3bf8f623-4d10-4fc5-ab4e-2bead3c4****   Pod "default/stress-demo-588f9646cf-7****" evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(76.72%)>threshold(50.00%)"
22s         Normal    EvictComplete      podmigrationjob/3bf8f623-4d10-4fc5-ab4e-2bead3c4****   Pod "default/stress-demo-588f9646cf-7****" has been evicted
55s         Normal    Descheduled        pod/stress-demo-588f9646cf-7****                       Pod evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(76.72%)>threshold(50.00%)"
55s         Normal    Killing            pod/stress-demo-588f9646cf-7****                       Stopping container stress

Step 1: Enable descheduling in ack-koordinator

• If the ack-koordinator component is not installed in the cluster, install the component and select Enable Descheduling For Ack-koordinator on the component configuration page. For more information, see Install the ack-koordinator component.

• If the ack-koordinator component is already installed in your cluster, on the component configuration page, select Enable Descheduling For Ack-koordinator. For the procedure, see Modify the ack-koordinator component.

Step 2: Enable the load hotspot descheduling plug-in

1. Create a `koord-descheduler-config.yaml` file using the following YAML content.

   The `koord-descheduler-config.yaml` file is a ConfigMap object that is used to enable the LowNodeLoad descheduling plug-in.

   Click to view the YAML file content

   # koord-descheduler-config.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: koord-descheduler-config
     namespace: kube-system
   data:
     koord-descheduler-config: |
       # The following content is the system configuration of koord-descheduler. Do not change the configuration.
       apiVersion: descheduler/v1alpha2
       kind: DeschedulerConfiguration
       leaderElection:
         resourceLock: leases
         resourceName: koord-descheduler
         resourceNamespace: kube-system
       deschedulingInterval: 120s # The execution interval. The descheduling plug-in runs every 120s.
       dryRun: false # The global read-only mode switch. If you enable this mode, Koordinator Descheduler does not perform any operations.
       # End of system configuration.
   
       profiles:
       - name: koord-descheduler
         plugins:
           balance:
             enabled:
               - name: LowNodeLoad # Enable the LowNodeLoad plug-in for load hotspot descheduling.
           evict:
             enabled:
               - name: MigrationController # Enable the eviction and migration controller.
   
         pluginConfig:
         - name: MigrationController # Parameters for descheduling and migration control.
           args:
             apiVersion: descheduler/v1alpha2
             kind: MigrationControllerArgs
             defaultJobMode: EvictDirectly
   
         - name: LowNodeLoad # Configuration of the LowNodeLoad plug-in.
           args:
             apiVersion: descheduler/v1alpha2
             kind: LowNodeLoadArgs
   
             lowThresholds:  # lowThresholds specifies the admission threshold for idle nodes. A node is considered idle if its usage of all resources is below the threshold.
               cpu: 20 # The CPU utilization is 20%.
               memory: 30  # The Memory utilization is 30%.
             highThresholds: # highThresholds specifies the admission threshold for hotspot nodes. A node is considered a hotspot if its usage of any resource is above the threshold.
               cpu: 50  # The CPU utilization is 50%.
               memory: 60 # The Memory utilization is 60%.
   
             evictableNamespaces: # The namespaces that can be descheduled. The include and exclude parameters are mutually exclusive. You can configure only one of them.
               include the following: # The include parameter specifies that only the following namespaces are processed.
                 - default
               # exclude: # The exclude parameter specifies the namespaces to be excluded.
                 # - "kube-system"
                 # - "koordinator-system"

2. Run the following command to apply the configuration to the cluster.

   kubectl apply -f koord-descheduler-config.yaml

3. Run the following command to restart the Koordinator Descheduler module.

   After you restart the Koordinator Descheduler module, the Koordinator Descheduler uses the most recently modified configuration.

   kubectl -n kube-system scale deploy ack-koord-descheduler --replicas 0
   deployment.apps/ack-koord-descheduler scaled
   kubectl -n kube-system scale deploy ack-koord-descheduler --replicas 1
   deployment.apps/ack-koord-descheduler scaled

Step 3 (Optional): Enable the scheduler load balancing plug-in

To enable the scheduler load balancing plug-in for optimal load balancing among nodes, see Step 1: Enable load-aware scheduling.

Step 4: Verify the descheduling feature

The following example uses a cluster that has three nodes, each with 104 cores and 396 GiB of memory.

1. Create a `stress-demo.yaml` file using the following YAML content.

   Click to view the YAML file content.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: stress-demo
     namespace: default
     labels:
       app: stress-demo
   spec:
     replicas: 6
     selector:
       matchLabels:
         app: stress-demo
     template:
       metadata:
         name: stress-demo
         labels:
           app: stress-demo
       spec:
         containers:
           - args:
               - '--vm'
               - '2'
               - '--vm-bytes'
               - '1600M'
               - '-c'
               - '2'
               - '--vm-hang'
               - '2'
             command:
               - stress
             image: polinux/stress
             imagePullPolicy: Always
             name: stress
             resources:
               limits:
                 cpu: '2'
                 memory: 4Gi
               requests:
                 cpu: '2'
                 memory: 4Gi
         restartPolicy: Always

2. Create the stress testing pod.

   kubectl create -f stress-demo.yaml
   deployment.apps/stress-demo created

3. Observe the status of the pod until it is running.

   kubectl get pod -o wide

   Expected output:

   NAME                           READY   STATUS    RESTARTS   AGE   IP            NODE                    NOMINATED NODE   READINESS GATES
   stress-demo-588f9646cf-s****   1/1     Running   0          82s   10.XX.XX.53   cn-beijing.10.XX.XX.53   <none>           <none>

   The output shows that the pod stress-demo-588f9646cf-s**** is scheduled to the node cn-beijing.10.XX.XX.53.

4. Increase the load level of the node cn-beijing.10.XX.XX.53 and then check the load of each node.

   kubectl top node

   Expected output:

   NAME                      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
   cn-beijing.10.XX.XX.215   17611m       17%    24358Mi         6%
   cn-beijing.10.XX.XX.53    63472m       63%    11969Mi         3%

   The output shows that the load of the node cn-beijing.10.XX.XX.53 is high at 63%, which exceeds the configured hot spot threshold of 50%. The load of the node cn-beijing.10.XX.XX.215 is low at 17%, which is below the configured idle threshold of 20%.

5. Enable load-aware hot spot descheduling. For more information, see Step 2: Enable the load-based hot spot descheduling plug-in.

6. Observe the pod changes.

   Wait for the descheduler to check for hot spot nodes and perform eviction and migration.

   Note

   By default, a node is identified as a hot spot if it exceeds the hot spot threshold for five consecutive checks, which takes 10 minutes.

   kubectl get pod -w

   Expected output:

   NAME                           READY   STATUS               RESTARTS   AGE     IP           NODE                     NOMINATED NODE   READINESS GATES
   stress-demo-588f9646cf-s****   1/1     Terminating          0          59s   10.XX.XX.53    cn-beijing.10.XX.XX.53     <none>           <none>
   stress-demo-588f9646cf-7****   1/1     ContainerCreating    0          10s   10.XX.XX.215   cn-beijing.10.XX.XX.215    <none>           <none>

7. Observe the event.

   kubectl get event | grep stress-demo-588f9646cf-s****

   Expected output:

   2m14s       Normal    Evicting            podmigrationjob/00fe88bd-8d4c-428d-b2a8-d15bcdeb****   Pod "default/stress-demo-588f9646cf-s****" evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(68.53%)>threshold(50.00%)"
   101s        Normal    EvictComplete       podmigrationjob/00fe88bd-8d4c-428d-b2a8-d15bcdeb****   Pod "default/stress-demo-588f9646cf-s****" has been evicted
   2m14s       Normal    Descheduled         pod/stress-demo-588f9646cf-s****                       Pod evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(68.53%)>threshold(50.00%)"
   2m14s       Normal    Killing             pod/stress-demo-588f9646cf-s****                       Stopping container stress

   The expected output is the migration record, which shows that the result is as expected. The pod on the hot spot node is descheduled to another node.

Advanced configuration

Advanced configuration parameters of the Koordinator Descheduler module

All parameter configurations for the Koordinator Descheduler are provided in a ConfigMap. The following shows the format of the advanced configuration parameters for load-based hot spot descheduling.

# koord-descheduler-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: koord-descheduler-config
  namespace: kube-system
data:
  koord-descheduler-config: |
    # The following content is the system configuration of koord-descheduler. Do not change the configuration.
    apiVersion: descheduler/v1alpha2
    kind: DeschedulerConfiguration
    leaderElection:
      resourceLock: leases
      resourceName: koord-descheduler
      resourceNamespace: kube-system
    deschedulingInterval: 120s # The execution interval. The descheduling plug-in runs every 120s.
    dryRun: false # The global read-only mode switch. If you enable this mode, koord-descheduler does not perform any operations.
    # End of system configuration.

    profiles:
    - name: koord-descheduler
      plugins:
        deschedule: 
          disabled:
            - name: "*" # All plug-ins are disabled by default. You do not need to explicitly configure this. This is for demonstration purposes only.
        balance:
          enabled:
            - name: LowNodeLoad # Enable the LowNodeLoad plug-in for load hotspot descheduling.
        evict:
          disabled:
            - name: "*" # All plug-ins are disabled by default. You do not need to explicitly configure this. This is for demonstration purposes only.
          enabled:
            - name: MigrationController # Enable the eviction and migration controller.

      pluginConfig:
      - name: MigrationController # Parameters for descheduling and migration control.
        args:
          apiVersion: descheduler/v1alpha2
          kind: MigrationControllerArgs
          defaultJobMode: EvictDirectly
          maxMigratingPerNode: 1 # The maximum number of pods that can be in a migrating state on each node.
          maxMigratingPerNamespace: 1  # The maximum number of pods that can be in a migrating state in each namespace.
          maxMigratingPerWorkload: 1 # The maximum number of pods that can be in a migrating state in each workload, such as a deployment.
          maxUnavailablePerWorkload: 2 # The maximum number of unavailable replicas for each workload, such as a deployment.
          evictLocalStoragePods: false # Specifies whether to allow pods configured with HostPath or EmptyDir to be descheduled.
          objectLimiters:
            workload: # Throttling for pod migration at the workload level. By default, a maximum of one replica can be migrated within 5 minutes after the first eviction.
              duration: 5m
              maxMigrating: 1

      - name: LowNodeLoad # Configuration of the LowNodeLoad plug-in.
        args:
          apiVersion: descheduler/v1alpha2
          kind: LowNodeLoadArgs

          lowThresholds:  # lowThresholds specifies the admission threshold for idle nodes. A node is considered idle if its usage of all resources is below the threshold.
            cpu: 20 # The CPU utilization is 20%.
            memory: 30  # The Memory utilization is 30%.
          highThresholds: # highThresholds specifies the admission threshold for hotspot nodes. A node is considered a hotspot if its usage of any resource is above the threshold.
            cpu: 50  # The CPU utilization is 50%.
            memory: 60 # The Memory utilization is 60%.

          anomalyCondition: # Hotspot node check configuration.
            consecutiveAbnormalities: 5 # A node is identified as a hotspot only if it exceeds highThresholds for multiple consecutive execution cycles. The counter is reset after the hotspot node is evicted.

          detectorCacheTimeout: "5m" # The timeout period for the anomaly check cache. Default value: 5 minutes. Make sure that this value is not less than the execution interval specified in deschedulingInterval.

          evictableNamespaces: # The namespaces that can be descheduled. The include and exclude parameters are mutually exclusive. You can configure only one of them.
            include the following: # The include parameter specifies that only the following namespaces are processed.
              - default
            # exclude: # The exclude parameter specifies the namespaces to be excluded.
              # - "kube-system"
              # - "koordinator-system"

          nodeSelector: # Process only the specified nodes.
            matchLabels:
              alibabacloud.com/nodepool-id: np77f520e1108f47559e63809713ce****

          podSelectors: # Process only some types of pods.
          - name: lsPods
            selector:
              matchLabels:
                koordinator.sh/qosClass: "LS"

Koordinator Descheduler system configuration

Eviction and migration control configuration

type MigrationObjectLimiter struct {
    Duration time.Duration `json:"duration,omitempty"`
    MaxMigrating *intstr.IntOrString `json:"maxMigrating,omitempty"`
}

objectLimiters:
  workload:
    duration: 5m
    maxMigrating: 1

LowNodeLoad plug-in configuration

highThresholds:
  cpu: 55 # The CPU utilization hotspot threshold is 55%.
  memory: 75 # The Memory utilization hotspot threshold is 75%.

lowThresholds:
  cpu: 25 # The CPU utilization idle threshold is 25%.
  memory: 25 # The Memory utilization idle threshold is 25%.

evictableNamespaces:
  exclude:
    - "kube-system"
    - "koordinator-system"

type PodSelector struct {
    name     string
    selector metav1.LabelSelector
}

# Process only LS-type pods.
podSelectors:
- name: lsPods
  selector:
    matchLabels:
      koordinator.sh/qosClass: "LS"

FAQ

What do I do if the node utilization reaches the threshold but pods on the node are not evicted?

This issue may occur for the following reasons. You can refer to the corresponding solutions to resolve the issue.

Why does the descheduler frequently restart?

The descheduler may frequently restart if its ConfigMap is invalid or does not exist. For more information, see Advanced configuration. Check the content and format of the ConfigMap, modify the ConfigMap, and then restart the descheduler. For more information about how to restart the descheduler, see Step 2: Enable the load-based hot spot descheduling plug-in.

How do I use load-aware scheduling and load hotspot descheduling together?

After you enable load-based hot spot descheduling, pods on hot spot nodes are evicted. The ACK scheduler selects appropriate nodes for pods that are created by upper-layer controllers, such as Deployments. To achieve optimal load balancing, we recommend that you enable load-aware scheduling at the same time. For more information, see Use load-aware scheduling.

We recommend that you set the loadAwareThreshold parameter for load-aware scheduling to the same value as the highThresholds parameter of the K8s Spot Rescheduler. For more information, see Scheduling policies. When the load of a node exceeds highThresholds, the K8s Spot Rescheduler evicts pods on that node. The scheduler then uses loadAwareThreshold to prevent new pods from being scheduled to the hot spot node. If you do not set the parameters to the same value, evicted pods may be rescheduled to the hot spot node. This issue is more likely to occur if a pod has a specified scope of schedulable nodes, but only a small number of nodes are available and the resource utilization of these nodes is similar.

What is the utilization algorithm that descheduling references?

The descheduler continuously monitors resource usage for a period and calculates an average value. A node is descheduled only if its average resource usage stays above a threshold for a certain period, which is 10 minutes by default. For memory, the descheduler's usage calculation excludes the page cache because the operating system can reclaim these resources. In contrast, the usage value returned by the kubectl top node command includes the page cache. You can use Managed Service for Prometheus to view the actual memory usage.