Configure ECI elastic scaling in ACK - Container Service for Kubernetes

Use tolerations and node affinity to schedule pods on Elastic Compute Service (ECS), ECI, or both with priority-based fallback.

Key concepts

Taint: A rule on a node that prevents certain pods from being scheduled to it.
Toleration: A setting on a pod that allows it to ignore a node's taints.
Node affinity: A pod property that attracts it to nodes with specific labels. Two types:
- requiredDuringSchedulingIgnoredDuringExecution: A hard requirement. Pods are scheduled only on matching nodes.
- preferredDuringSchedulingIgnoredDuringExecution: A soft preference. The scheduler tries to match the rule but schedules the pod elsewhere if no match is found.

See Taints and Tolerations and Node affinity.

Prerequisites

An ACK managed Pro cluster running Kubernetes v1.22 or later.
Version 2.10.0 or later of the ack-virtual-node add-on (see Virtual nodes).
Version 5.9 or later of the kube-scheduler component, with the virtual node scheduling policy enabled.
On the Add-ons page, find Kube Scheduler, click Configuration, and confirm that Enable Virtual Node-based Pod Scheduling is selected.

Configuration examples

Virtual nodes in ACK clusters have the taint virtual-kubelet.io/provider=alibabacloud:NoSchedule by default. To schedule pods on virtual nodes (ECI), add a toleration for this taint.

Example:

      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: alibabacloud
        effect: NoSchedule

Combine tolerations and node affinity to create scheduling policies for different resource needs.

Prefer ECS: Use ECS by default; fall back to ECI when ECS capacity is insufficient.
Use only ECI: Use only ECI resources.
Use only ECS: Use only existing ECS resources.

Prefer ECS

Add a toleration for the default taint, then use preferredDuringSchedulingIgnoredDuringExecution to prefer nodes without the type: virtual-kubelet label (ECS nodes).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ecs-prefer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: type
                operator: NotIn
                values:
                - virtual-kubelet
      containers:
      - name: my-container
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6

Note

About node affinity:

Multiple nodeSelectorTerms are combined with OR — a pod is scheduled if any matchExpressions match. Multiple expressions within one matchExpressions are combined with AND — all must match.
preferredDuringSchedulingIgnoredDuringExecution does not guarantee ECI-only fallback — pods may still land on ECI when ECS capacity is sufficient. For stricter control, combine with requiredDuringSchedulingIgnoredDuringExecution.

The following YAML prefers ECS nodes with a specific label (such as label_1=key_1) and falls back to virtual nodes when capacity is needed.

View YAML details

apiVersion: apps/v1
kind: Deployment
metadata:
  name: some-ecs-prefer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
      affinity:
        nodeAffinity:
    # The pod must be scheduled to a node with the label_1:key_1 or type:virtual-kubelet label.
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: label_1
                operator: In
                values:
                - key_1
            - matchExpressions:
              - key: type
                operator: In
                values:
                - virtual-kubelet
    # The pod is preferentially scheduled to a node with the label_1:key_1 label, and secondarily to a node with the type:virtual-kubelet label.
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - key: label_1
                operator: In
                values:
                - key_1
          - weight: 1
            preference:
              matchExpressions:
              - key: type
                operator: In
                values:
                - virtual-kubelet
      containers:
      - name: my-container
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6

Use only ECI

Add a toleration for the default virtual node taint, then use requiredDuringSchedulingIgnoredDuringExecution to require nodes with the type: virtual-kubelet label.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: eci-only
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: In
                values:
                - virtual-kubelet
      containers:
      - name: my-container
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6

Use only ECS

By default, pods are scheduled only on ECS nodes because virtual nodes have a taint. No extra configuration is needed.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ecs-only
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-container
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6