Configure resource allocation between ECS and ECI using taints and node affinity - Container Service for Kubernetes

Concepts

Taints: Rules set on a node that prevent pods from being scheduled to it, unless the pod explicitly tolerates the taint.
Tolerations: Pod-level properties that allow a pod to be scheduled to nodes with a matching taint.
Node affinity: A scheduling constraint that targets specific nodes based on their labels. Two types are available:
- requiredDuringSchedulingIgnoredDuringExecution: A hard requirement. The pod is scheduled only to nodes that satisfy the rule.
- preferredDuringSchedulingIgnoredDuringExecution: A soft preference. The scheduler favors nodes that satisfy the rule but can fall back to others.

For more information, see Taints and tolerations and Node affinity.

Prerequisites

Before you begin, ensure that you have:

An ACK Managed Cluster Pro Edition running version v1.22 or later
The ack-virtual-node component version v2.10.0 or later deployed in the cluster. For installation and upgrade steps, see Virtual nodes.
The kube-scheduler component version v5.9 or later, with virtual node scheduling enabled. To verify, go to the Add-ons page, find Kube Scheduler, click Configuration, and confirm that Enable Virtual Node-based Pod Scheduling is selected.

How it works

Virtual nodes in an ACK cluster carry a default taint:

virtual-kubelet.io/provider=alibabacloud:NoSchedule

This taint prevents pods from landing on virtual nodes unless you explicitly add a toleration. Three scheduling strategies are available:

Strategy	How it works	When to use
Prioritize ECS	Pods go to ECS nodes first; virtual nodes serve as overflow	Variable workloads that need elastic capacity
Use only ECI	Pods run exclusively on virtual nodes backed by ECI	Batch jobs, short-lived tasks, or serverless workloads
Use only ECS	Pods run on ECS nodes only; no ECI is used	Workloads with strict requirements on ECS-only infrastructure

Configuration examples

Prioritize ECS

This strategy uses a soft preference (preferredDuringSchedulingIgnoredDuringExecution) to favor ECS nodes. When ECS capacity is exhausted, the scheduler places pods on virtual nodes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ecs-prefer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: virtual-kubelet.io/provider   # Allow scheduling to virtual nodes
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: type
                operator: NotIn          # Prefer nodes that are NOT virtual nodes (i.e., ECS)
                values:
                - virtual-kubelet
      containers:
      - name: my-container
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6

preferredDuringSchedulingIgnoredDuringExecution does not guarantee that pods land on ECI only when ECS is fully exhausted. The scheduler uses the preference as a scoring factor, so pods may occasionally land on virtual nodes even when ECS nodes have available capacity. For stricter placement control, combine this with requiredDuringSchedulingIgnoredDuringExecution rules.

Prioritize a specific ECS node group, with ECI as overflow

To prefer ECS nodes with a specific label (for example, label_1=key_1) and allow virtual nodes only as overflow, combine required and preferred rules:

Expand to view YAML details

apiVersion: apps/v1
kind: Deployment
metadata:
  name: some-ecs-prefer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: virtual-kubelet.io/provider   # Allow scheduling to virtual nodes
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
      affinity:
        nodeAffinity:
          # Hard requirement: pod must land on a node with label_1=key_1 OR type=virtual-kubelet.
          # Multiple terms under nodeSelectorTerms are evaluated with logical OR.
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: label_1
                operator: In
                values:
                - key_1
            - matchExpressions:
              - key: type
                operator: In
                values:
                - virtual-kubelet
          # Soft preference: weight 100 strongly favors label_1=key_1 nodes (ECS),
          # weight 1 as fallback for virtual nodes (ECI).
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - key: label_1
                operator: In
                values:
                - key_1
          - weight: 1
            preference:
              matchExpressions:
              - key: type
                operator: In
                values:
                - virtual-kubelet
      containers:
      - name: my-container
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6

`matchExpressions` logic

Multiple terms under nodeSelectorTerms use logical OR — the pod is scheduled if any term matches.
Multiple expressions within a single matchExpressions block use logical AND — all expressions must match.

Operator reference

Operator	Behavior
`In`	Node label value is in the specified list
`NotIn`	Node label value is not in the specified list
`Exists`	Node has the specified label key (any value)
`DoesNotExist`	Node does not have the specified label key

Use only ECI

This strategy uses a hard requirement (requiredDuringSchedulingIgnoredDuringExecution) to restrict pods to virtual nodes exclusively.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: eci-only
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: virtual-kubelet.io/provider   # Allow scheduling to virtual nodes
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
      affinity:
        nodeAffinity:
          # Hard requirement: pod must be scheduled to a node with type=virtual-kubelet.
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: In             # Only nodes with type=virtual-kubelet qualify
                values:
                - virtual-kubelet
      containers:
      - name: my-container
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6

Use only ECS

Because virtual nodes carry the virtual-kubelet.io/provider=alibabacloud:NoSchedule taint by default, pods without a matching toleration are automatically restricted to ECS nodes. No affinity configuration is needed.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ecs-only
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      # No toleration configured — the default taint on virtual nodes
      # prevents pods from landing there.
      containers:
      - name: my-container
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6