Container Service for Kubernetes:Spread ECI pods across zones and configure affinity scheduling

Prerequisites

Before you begin, make sure you have:

• An ACK managed Pro cluster that meets the following requirements:

  • Kubernetes version 1.22 or later

  • ACK Virtual Node component version 2.10.0 or later

  • kube-scheduler component version 5.9 or later, with the virtual node-based pod scheduling policy enabled

• Multiple zones (vSwitches) configured in the eci-profile so pods can be scheduled across zones

• The nodeAffinity, podAffinity, or topologySpreadConstraints fields set in your pod spec, or a ResourcePolicy configured for the pod

Usage notes

• Set topologyKey to topology.kubernetes.io/zone.

• The scheduling behavior described in this topic does not apply when:

  • The k8s.aliyun.com/eci-schedule-strategy: "VSwitchOrdered" annotation is set on the pod (multi-zone scheduling follows a fixed vSwitch order instead).

  • The k8s.aliyun.com/eci-fail-strategy: "fail-fast" annotation is set on the pod.

Choose an approach

Two Kubernetes mechanisms serve different goals:

Both examples below use the same ECI-required fields: a toleration for virtual-kubelet.io/provider and a preferredDuringSchedulingIgnoredDuringExecution node affinity that prefers Elastic Compute Service (ECS) nodes over virtual nodes. The only thing that changes is the spread or affinity rule itself.

Example 1: Spread pods evenly across zones

This example creates a Deployment with 10 replicas distributed evenly across all available zones.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: with-pod-topology-spread
  labels:
    app: with-pod-topology-spread
spec:
  replicas: 10
  selector:
    matchLabels:
      app: with-pod-topology-spread
  template:
    metadata:
      labels:
        app: with-pod-topology-spread
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: type
                operator: NotIn
                values:
                - virtual-kubelet
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: with-pod-topology-spread
      tolerations:
        - key: "virtual-kubelet.io/provider"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
      - name: with-pod-topology-spread
        image: registry.k8s.io/pause:2.0
        resources:
          requests:
            cpu: "1"
            memory: "256Mi"

kubectl get po -lapp=with-pod-topology-spread \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName \
  --no-headers | grep -v "<none>"

kubectl get po -lapp=with-pod-topology-spread \
  -o custom-columns=NODE:.spec.nodeName \
  --no-headers | grep -v "<none>" \
  | xargs -I {} kubectl get no {} -o json \
  | jq '.metadata.labels["topology.kubernetes.io/zone"]' \
  | sort | uniq -c

Example 2: Deploy pods in a specific zone

This example creates a Deployment with 3 replicas that must all land in the same zone. Use this pattern when low inter-pod latency matters more than spreading across zones.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: with-affinity
  labels:
    app: with-affinity
spec:
  replicas: 3
  selector:
    matchLabels:
      app: with-affinity
  template:
    metadata:
      labels:
        app: with-affinity
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - with-affinity
            topologyKey: topology.kubernetes.io/zone
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: type
                operator: NotIn
                values:
                - virtual-kubelet
      tolerations:
        - key: "virtual-kubelet.io/provider"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
      - name: with-affinity
        image: registry.k8s.io/pause:2.0

requiredDuringSchedulingIgnoredDuringExecution:
  nodeSelectorTerms:
  - matchExpressions:
    - key: topology.kubernetes.io/zone
      operator: In
      values:
      - cn-beijing-a

kubectl get po -lapp=with-affinity \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName \
  --no-headers | grep -v "<none>"

kubectl get po -lapp=with-affinity \
  -o custom-columns=NODE:.spec.nodeName \
  --no-headers | grep -v "<none>" \
  | xargs -I {} kubectl get no {} -o json \
  | jq '.metadata.labels["topology.kubernetes.io/zone"]' \
  | sort | uniq -c

Strict ECI pod topology spread

By default, kube-scheduler targets an even zone distribution but does not block pod placement when ECI creation fails in some zones. This can silently violate the maxSkew constraint.

For example, with maxSkew: 1 and three zones (A, B, C), kube-scheduler evenly deploys the pods of a workload across all zones. If ECI creation fails in Zone B and Zone C, pods only run in Zone A — violating the constraint specified by maxSkew.

Enable strict ECI pod topology spread to guarantee the constraint is honored. With strict mode on, kube-scheduler first dispatches a pod to each zone and holds back pending pods until the scheduled pods are created. The figure below shows the initial state: one pod dispatched to each zone, all others pending.

Even after Pod A1 is created, kube-scheduler does not schedule the next pod — because if Zone B or Zone C fails, the constraint would be violated. Only after Pod B1 is also created does kube-scheduler schedule a pod to Zone C. Pods with green shading indicate created pods.

To disable strict spread and allow kube-scheduler to schedule pods regardless of constraint satisfaction, set whenUnsatisfiable: ScheduleAnyway. For parameter details, see Spread constraint definition.