Elastic Container Instance:Spread Elastic Container Instance-based pods across zones and configure affinities

Deploy a Deployment with zone spread

1. Create a file named deployment.yaml with the following content. This creates a 10-replica Deployment with a maximum skew of 1 across zones. The following optional parameters are also available in topologySpreadConstraints:

   topologySpreadConstraints:
     - maxSkew: <integer>
       minDomains: <integer>         # Optional. Beta in Kubernetes 1.25 and later.
       topologyKey: <string>
       whenUnsatisfiable: <string>
       labelSelector: <object>
       matchLabelKeys: <list>        # Optional. Beta in Kubernetes 1.27 and later.
       nodeAffinityPolicy: [Honor|Ignore]  # Optional. Beta in Kubernetes 1.26 and later.
       nodeTaintsPolicy: [Honor|Ignore]    # Optional. Beta in Kubernetes 1.26 and later.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: with-pod-topology-spread
     labels:
       app: with-pod-topology-spread
   spec:
     replicas: 10
     selector:
       matchLabels:
         app: with-pod-topology-spread
     template:
       metadata:
         labels:
           app: with-pod-topology-spread
       spec:
         topologySpreadConstraints:
           - maxSkew: 1
             topologyKey: topology.kubernetes.io/zone
             whenUnsatisfiable: DoNotSchedule
             labelSelector:
               matchLabels:
                 app: with-pod-topology-spread
         containers:
         - name: with-pod-topology-spread
           image: registry.k8s.io/pause:2.0
           resources:
             requests:
               cpu: "1"
               memory: "256Mi"

2. Apply the manifest:

   kubectl apply -f deployment.yaml

3. Verify the scheduling result. Check which nodes the pods are running on:

   kubectl get po -lapp=with-pod-topology-spread -ocustom-columns=NAME:.metadata.name,NODE:.spec.nodeName --no-headers | grep -v "<none>"

   Check how many pods landed in each zone:

   kubectl get po -lapp=with-pod-topology-spread -ocustom-columns=NODE:.spec.nodeName --no-headers | grep -v "<none>" | xargs -I {} kubectl get no {} -ojson | jq '.metadata.labels["topology.kubernetes.io/zone"]' | sort | uniq -c

How strict topology spread works

By default, the scheduler counts running pods when computing the spread—it does not wait for pending pods to be created. If ECI fails to create pods in some zones, the spread constraint based on maxSkew may be violated.

For example, suppose the scheduler places pods in Zone A, Zone B, and Zone C. If pod creation fails in Zone B and Zone C, Zone A ends up with more pods than allowed by maxSkew: 1, and no Elastic Container Instance-based pod runs in Zone B or Zone C. This violates the constraint specified by the maxSkew parameter.

ACK Serverless Pro clusters support strict topology spread to prevent this. With strict topology spread enabled, the scheduler first schedules a pod to each zone, and does not schedule pending pods until the scheduled pods are created:

The scheduler first places one pod in each zone. It holds subsequent pods until the initial set is created:

For instance, even after Pod A1 is created, pending pods are not scheduled. This is because if the pod in Zone B or Zone C fails to be created, the constraint specified by the maxSkew parameter is violated. After Pod B1 is created, kube-scheduler schedules a pod to Zone C. Pods with green shading are created.

To disable strict topology spread, set whenUnsatisfiable: ScheduleAnyway. For details, see Spread constraint definition.

Pin pods to a specific zone (nodeAffinity)

To target a specific zone, use nodeAffinity and specify the zone label value. The following example pins all pods to Beijing Zone A (cn-beijing-a).

1. Create a file named deployment.yaml with the following content:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: with-affinity
     labels:
       app: with-affinity
   spec:
     replicas: 3
     selector:
       matchLabels:
         app: with-affinity
     template:
       metadata:
         labels:
           app: with-affinity
       spec:
         affinity:
           nodeAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
               nodeSelectorTerms:
               - matchExpressions:
                 - key: topology.kubernetes.io/zone
                   operator: In
                   values:
                   - cn-beijing-a
         containers:
         - name: with-affinity
           image: registry.k8s.io/pause:2.0

2. Apply the manifest:

   kubectl apply -f deployment.yaml

3. Verify the scheduling result. Check which nodes the pods are running on:

   kubectl get po -lapp=with-affinity -ocustom-columns=NAME:.metadata.name,NODE:.spec.nodeName --no-headers | grep -v "<none>"

   Check the zone distribution:

   kubectl get po -lapp=with-affinity -ocustom-columns=NODE:.spec.nodeName --no-headers | grep -v "<none>" | xargs -I {} kubectl get no {} -ojson | jq '.metadata.labels["topology.kubernetes.io/zone"]' | sort | uniq -c