Container Service for Kubernetes:Enable the descheduling feature

Configure the plug-in

Create a file named koord-descheduler-config.yaml with the following content. This ConfigMap enables RemovePodsViolatingNodeTaints and configures it to ignore the deschedule=not-allow taint.

# koord-descheduler-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: koord-descheduler-config
  namespace: kube-system
data:
  koord-descheduler-config: |
    # Do not modify the following system configuration of koord-desheduler.
    apiVersion: descheduler/v1alpha2
    kind: DeschedulerConfiguration
    leaderElection:
      resourceLock: leases
      resourceName: koord-descheduler
      resourceNamespace: kube-system
    deschedulingInterval: 120s # The interval at which the descheduler runs. Set to 120 seconds here.
    dryRun: false # The global read-only mode. After you enable this mode, koord-descheduler does not perform any operations.
    # The preceding configuration is the system configuration.

    profiles:
    - name: koord-descheduler
      plugins:
        deschedule:
          enabled:
            - name: RemovePodsViolatingNodeTaints  # Enable the node taint verification plug-in.

      pluginConfig:
      - name: RemovePodsViolatingNodeTaints # Configure the node taint verification plug-in.
        args:
          excludedTaints:
          - deschedule=not-allow # Ignore nodes whose taint key is deschedule and taint value is not-allow.

      # Required for RemovePodsViolatingNodeTaints to take effect. Do not remove.
      - name: MigrationController # Configure the migration controller.
        args:
          apiVersion: descheduler/v1alpha2
          kind: MigrationControllerArgs
          defaultJobMode: EvictDirectly

RemovePodsViolatingNodeTaints parameters:

Apply the configuration

1. Apply the ConfigMap to the cluster:

   kubectl apply -f koord-descheduler-config.yaml

2. Restart Koordinator Descheduler to pick up the new configuration:

   kubectl -n kube-system scale deploy ack-koord-descheduler --replicas 0
   # Expected output:
   # deployment.apps/ack-koord-descheduler scaled
   kubectl -n kube-system scale deploy ack-koord-descheduler --replicas 1
   # Expected output:
   # deployment.apps/ack-koord-descheduler scaled

Example of advanced configurations

The following example shows a fully configured Koordinator Descheduler ConfigMap. It uses DeschedulerConfiguration for global settings, enables RemovePodsViolatingNodeTaints as the descheduling policy, and uses MigrationController as the evictor.

# koord-descheduler-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: koord-descheduler-config
  namespace: kube-system
data:
  koord-descheduler-config: |
    # Do not modify the following system configuration of koord-desheduler.
    apiVersion: descheduler/v1alpha2
    kind: DeschedulerConfiguration
    leaderElection:
      resourceLock: leases
      resourceName: koord-descheduler
      resourceNamespace: kube-system
    dryRun: false # The global read-only mode. After you enable this mode, koord-descheduler does not perform any operations.
    deschedulingInterval: 120s # The interval at which the descheduler runs. The interval is set to 120 seconds in this example.
    nodeSelector: # The nodes that are involved in descheduling. By default, all nodes are descheduled.
      matchLabels:
        alibabacloud.com/nodepool-id: nodepool-1 # Configure it based on your requirements.
    maxNoOfPodsToEvictPerNode: 10 # The maximum number of pods that can be evicted from a node. The limit takes effect on a global scale. By default, no limit is configured.
    maxNoOfPodsToEvictPerNamespace: 10 # The maximum number of pods that can be evicted from a namespace. The limit takes effect on a global scale. By default, no limit is configured.
    # The preceding configuration is the system configuration.

    # The template list.
    profiles:
    - name: koord-descheduler # The name of the template.

      # Scope: apply this template only to the specified nodes.
      # Method 1: Select nodes in one node pool.
      nodeSelector:
        matchLabels:
          alibabacloud.com/nodepool-id: nodepool-1 # Configure it based on your requirements
      # Method 2: Select nodes in multiple node pools.
      # nodeSelector:
      #   matchExpressions:
      #   - key: alibabacloud.com/nodepool-id
      #     operator: In
      #     values:
      #     - nodepool-1
      #     - nodepool-2

      plugins:
        deschedule: # All plug-ins are disabled by default. Specify the ones to enable.
          enabled:
            - name: RemovePodsViolatingNodeTaints  # Enable the node taint verification plug-in.
        balance: # All plug-ins are disabled by default.
          disabled:
            - name: "*" # Disable all Balance plug-ins.
        evict:
          enabled:
            - name: MigrationController # MigrationController is enabled by default.
        filter:
          enabled:
            - name: MigrationController # Use MigrationController's filtering policy by default.

      pluginConfig:
      - name: RemovePodsViolatingNodeTaints
        args:
          excludedTaints:
          - deschedule=not-allow # Ignore nodes whose taint key is deschedule and taint value is not-allow.
          - reserved # Ignore nodes whose taint key is reserved.
          includePreferNoSchedule: false # When false, only checks NoSchedule taints.
          namespaces:
            include: # Restrict descheduling to these namespaces.
              - "namespace1"
              - "namespace2"
            # exclude: # Alternatively, exclude these namespaces.
            #   - "namespace1"
            #   - "namespace2"
          labelSelector: # Only deschedule pods matching these labels.
            accelerator: nvidia-tesla-p100

      - name: MigrationController
        args:
          apiVersion: descheduler/v1alpha2
          kind: MigrationControllerArgs
          defaultJobMode: EvictDirectly
          evictLocalStoragePods: false # When false, pods using emptyDir or hostPath are not descheduled.
          maxMigratingPerNode: 1 # Maximum pods migrated simultaneously on a node.
          maxMigratingPerNamespace: 1  # Maximum pods migrated simultaneously in a namespace.
          maxMigratingPerWorkload: 1 # Maximum pods migrated simultaneously in a workload.
          maxUnavailablePerWorkload: 2 # Maximum unavailable replicated pods allowed in a workload.
          objectLimiters:
            workload: # Throttle workload-level migration. Default: only 1 pod per workload within 5 minutes.
              duration: 5m
              maxMigrating: 1
          evictionPolicy: Eviction # Use the Eviction API by default.

System configurations

Configure global, system-level behavior in DeschedulerConfiguration.

Template configurations

Koordinator Descheduler uses templates (profiles) to group descheduling policies and evictors. Each template has the following fields:

• `name`: A string identifier for the template.

• `plugins`: Enables or disables descheduling policies (deschedule, balance), evictors (evict), and pre-eviction filters (filter).

• `pluginConfig`: Per-plug-in advanced arguments. Set the name field to match the plug-in name and configure args. See Configure policy plug-ins and Configure evictor plug-ins.

• `nodeSelector`: Limits the template scope to specific nodes. If not set, the template applies to all nodes.

The nodeSelector field in template configurations requires ack-koordinator v1.6.1-ack.1.16 or later.

`plugins` field reference:

Configure policy plug-ins

Koordinator Descheduler supports six Deschedule plug-ins and five Balance plug-ins from Kubernetes Descheduler. The LowNodeLoad plug-in is provided by Koordinator. For more information, see Work with load-aware hotspot descheduling.

Configure evictor plug-ins

Koordinator Descheduler supports two evictor plug-ins: DefaultEvictor and MigrationController.