All Products
Search

This Product

    Container Service for Kubernetes:Spread Elastic Container Instance-based pods across zones and configure affinities

    Document Center

    Container Service for Kubernetes:Spread Elastic Container Instance-based pods across zones and configure affinities

    Last Updated:Nov 17, 2023

    High availability and high performance are essential to distributed jobs. In an ACK Serverless Pro cluster, you can spread distributed jobs across zones based on Kubernetes-native scheduling semantics. You can also configure affinities to deploy distributed jobs in specific zones based on Kubernetes-native scheduling semantics. This improves the efficiency of job deployment. This topic describes how to spread Elastic Container Instance-based pods across zones and configure affinities.

    Important

    To spread Elastic Container Instance-based pods across zones, submit a ticket to apply to be added to the whitelist.

    Table of contents

    Prerequisites

    • An ACK Serverless Pro cluster is created and the Kubernetes version of the cluster is 1.22 or later. For more information, see Create an ACK Serverless cluster.

    • The kube-scheduler component is installed. For more information, see kube-scheduler.

    • When you schedule Elastic Container Instance-based pods, make sure that you have specified the desired vSwitches in Elastic Container Instance Profile.

    • Elastic Container Instance-based pods can be spread across zones only when the nodeAffinity, podAffinity, tolerations, and topologySpreadConstraints parameters are configured for the pods or the pods match an existing resource policy.

    Precautions

    • When you spread Elastic Container Instance-based pods across zones, you can set the topologyKey parameter only to topology.kubernetes.io/zone.

    • When you spread Elastic Container Instance-based pods across zones, you cannot add annotations to specify the priorities of the vSwitches to which pods are spread. If you have specified the priorities of vSwitches to which a pod can be spread, the pod cannot be spread across zones.

    • You cannot spread Elastic Container Instance-based pods across zones in FastFailed mode. If the FastFailed mode is enabled for a pod, the pod cannot be spread across zones.

    Terms

    Spread Elastic Container Instance-based pods across zones and configure affinities

    The following examples show how to spread Elastic Container Instance-based pods across zones and configure affinities in an ACK Serverless Pro cluster that runs Kubernetes 1.22.

    Example 1: Use topology spread constraints to spread Elastic Container Instance-based pods across zones

    1. Add a topology spread constraint to the configuration of a workload.

      Perform the following steps to specify a topology spread constraint in the Spec parameter in the configuration of a pod or the Spec parameter in the configuration of a workload, such as a Deployment or Job. 

        topologySpreadConstraints:
    - maxSkew: <integer>
      minDomains: <integer> # This parameter is optional and is in the Beta phase in Kubernetes 1.25 and later. 
      topologyKey: <string>
      whenUnsatisfiable: <string>
      labelSelector: <object>
      matchLabelKeys: <list> # This parameter is optional and is in the Beta phase in Kubernetes 1.27 and later. 
      nodeAffinityPolicy: [Honor|Ignore] # This parameter is optional and is in the Beta phase in Kubernetes 1.26 and later. 
      nodeTaintsPolicy: [Honor|Ignore] # This parameter is optional and is in the Beta phase in Kubernetes 1.26 and later.

      In this example, a Deployment whose pods are evenly distributed to multiple zones is created. For more information about the parameters, see topologySpreadConstraints field. The following code block shows the YAML template of the Deployment:

      Show YAML content

      apiVersion: apps/v1
kind: Deployment
metadata:
  name: with-pod-topology-spread
  labels:
    app: with-pod-topology-spread
spec:
  replicas: 10
  selector:
    matchLabels:
      app: with-pod-topology-spread
  template:
    metadata:
      labels:
        app: with-pod-topology-spread
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: with-pod-topology-spread
      containers:
      - name: with-pod-topology-spread
        image: registry.k8s.io/pause:2.0
        resources:
          requests:
            cpu: "1"
            memory: "256Mi"

    2. Create a workload.

      Create a file named deployment.yaml and copy the preceding YAML template to the file. Then, run the following command to create a Deployment in the cluster: 

      kubectl apply -f deployment.yaml

    3. Verify the scheduling result of the workload.

      • Run the following command to query the nodes on which the Deployment deploys the pods: 

        kubectl get po -lapp=with-pod-topology-spread -ocustom-columns=NAME:.metadata.name,NODE:.spec.nodeName --no-headers | grep -v "<none>"

      • Run the following command to query the number of pods that are created by the Deployment in each zone: 

        kubectl get po -lapp=with-pod-topology-spread -ocustom-columns=NODE:.spec.nodeName --no-headers | grep -v "<none>" | xargs -I {} kubectl get no {} -ojson | jq '.metadata.labels["topology.kubernetes.io/zone"]' | sort | uniq -c

    Example 2: Configure pod affinities and node affinities to deploy pods in specific zones

    1. Add affinities to the configuration of a workload.

      In this example, a Deployment whose pods are deployed in a single zone is created. For more information about the parameters, see Node affinity. The following code block shows the YAML template of the Deployment:

      Show YAML content

      apiVersion: apps/v1
kind: Deployment
metadata:
  name: with-affinity
  labels:
    app: with-affinity
spec:
  replicas: 3
  selector:
    matchLabels:
      app: with-affinity
  template:
    metadata:
      labels:
        app: with-affinity
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - with-affinity
            topologyKey: topology.kubernetes.io/zone
      containers:
      - name: with-affinity
        image: registry.k8s.io/pause:2.0

      If you want to deploy the pods in a specific zone, delete the podAffinity parameter and add the following constraint to the nodeAffinity parameter: The following configuration specifies that the pods must be deployed in Beijing Zone A. 

      requiredDuringSchedulingIgnoredDuringExecution:
  nodeSelectorTerms:
  - matchExpressions:
    - key: topology.kubernetes.io/zone
      operator: In
      values:
      - cn-beijing-a

      The following code block shows the sample code that contains the nodeAffinity parameter. Pods are deployed only in Beijing Zone A.

      Show YAML content

      apiVersion: apps/v1
kind: Deployment
metadata:
  name: with-affinity
  labels:
    app: with-affinity
spec:
  replicas: 3
  selector:
    matchLabels:
      app: with-affinity
  template:
    metadata:
      labels:
        app: with-affinity
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - cn-beijing-a
      containers:
      - name: with-affinity
        image: registry.k8s.io/pause:2.0

    2. Create a workload.

      Create a file named deployment.yaml and copy the preceding YAML template to the file. Then, run the following command to create a Deployment in the cluster: 

      kubectl apply -f deployment.yaml

    3. Verify the scheduling result of the workload.

      • Run the following command to query the nodes on which the Deployment deploys the pods: 

        kubectl get po -lapp=with-affinity -ocustom-columns=NAME:.metadata.name,NODE:.spec.nodeName --no-headers | grep -v "<none>"

      • Run the following command to query the number of pods that are created by the Deployment in each zone: 

        kubectl get po -lapp=with-affinity -ocustom-columns=NODE:.spec.nodeName --no-headers | grep -v "<none>" | xargs -I {} kubectl get no {} -ojson | jq '.metadata.labels["topology.kubernetes.io/zone"]' | sort | uniq -c

    Strict Elastic Container Instance-based pod topology spread

    By default, if you force the system to spread Elastic Container Instance-based pods across zones, kube-scheduler evenly deploys the pods of a workload across all zones. However, Elastic Container Instance-based pods may fail to be created in some zones. The following figure shows the scheduling result when the maxSkew parameter is set to 1. For more information about maxSkew, see maxSkew.1..png

    If the Elastic Container Instance-based pods in Zone B and Zone C fail to be created, two Elastic Container Instance-based pods will run in Zone A, whereas no Elastic Container Instance-based pod runs in Zone B or Zone C. This violates the constraint specified by the maxSkew parameter.

    In an ACK Serverless Pro cluster, you can enable strict Elastic Container Instance-based pod topology spread to ensure that pods are strictly spread across zones. After you enable strict Elastic Container Instance-based pod topology spread, kube-scheduler first schedules a pod to each of Zone A, Zone B, and Zone C. kube-scheduler does not schedule pending pods until the scheduled pods are successfully created, as shown in the following figure.2..png

    Even if Pod A1 is successfully created, pending pods are not scheduled. This is because if the pod in Zone B or Zone C fails to be created, the constraint specified by the maxSkew parameter is violated. After Pod B1 is successfully created, kube-scheduler schedules a pod to Zone C. The following figure shows the scheduling result. Pods with orange shading are successfully created.3..png

    If you want to disable strict Elastic Container Instance-based pod topology spread, set the whenUnsatisfiable parameter to ScheduleAnyway. For more information, see Spread constraint definition.