Container Service for Kubernetes:Use the ack-co-scheduler component to enable co-scheduling

Submit a TensorFlow distributed job

The following example submits a TensorFlow distributed training job with Gang Scheduling enabled. Both the PS and Worker pods use pod-group.scheduling.sigs.k8s.io labels to form a pod group.

apiVersion: "kubeflow.org/v1"
kind: "TFJob"
metadata:
  name: "tf-smoke-gpu"
spec:
  tfReplicaSpecs:
    PS:
      replicas: 1
      template:
        metadata:
          creationTimestamp: null
          labels:
            pod-group.scheduling.sigs.k8s.io/name: tf-smoke-gpu
            pod-group.scheduling.sigs.k8s.io/min-available: "2"
        spec:
          schedulerName: ack-co-scheduler   # Route pods through the ACK co-scheduler.
          containers:
          - args:
            - python
            - tf_cnn_benchmarks.py
            - --batch_size=32
            - --model=resnet50
            - --variable_update=parameter_server
            - --flush_stdout=true
            - --num_gpus=1
            - --local_parameter_device=cpu
            - --device=cpu
            - --data_format=NHWC
            image: registry.cn-hangzhou.aliyuncs.com/kubeflow-images-public/tf-benchmarks-cpu:v20171202-bdab599-dirty-284af3
            name: tensorflow
            ports:
            - containerPort: 2222
              name: tfjob-port
            resources:
              limits:
                cpu: '10'
            workingDir: /opt/tf-benchmarks/scripts/tf_cnn_benchmarks
          restartPolicy: OnFailure
    Worker:
      replicas: 4
      template:
        metadata:
          creationTimestamp: null
          labels:
            pod-group.scheduling.sigs.k8s.io/name: tf-smoke-gpu
            pod-group.scheduling.sigs.k8s.io/min-available: "2"
        spec:
          schedulerName: ack-co-scheduler
          containers:
          - args:
            - python
            - tf_cnn_benchmarks.py
            - --batch_size=32
            - --model=resnet50
            - --variable_update=parameter_server
            - --flush_stdout=true
            - --num_gpus=1
            - --local_parameter_device=cpu
            - --device=gpu
            - --data_format=NHWC
            image: registry.cn-hangzhou.aliyuncs.com/kubeflow-images-public/tf-benchmarks-cpu:v20171202-bdab599-dirty-284af3
            name: tensorflow
            ports:
            - containerPort: 2222
              name: tfjob-port
            resources:
              limits:
                cpu: 10
            workingDir: /opt/tf-benchmarks/scripts/tf_cnn_benchmarks
          restartPolicy: OnFailure

Key fields:

Verify Gang scheduling

After submitting the job, check that pods are entering a pending state together:

kubectl get pods -l pod-group.scheduling.sigs.k8s.io/name=tf-smoke-gpu

Pods remain in Pending until the scheduler can place at least min-available pods simultaneously. This is expected behavior, not an error. If pods stay pending for an extended period, run the following command and check the Events section for scheduling messages:

kubectl describe pod <pod-name>

For more information, see Use Gang scheduling.

Prerequisites

Deploy the resource-controller component before enabling this feature. See Manage add-ons.

Enable CPU topology-aware scheduling

Add the cpuset-scheduler: "true" annotation to your Deployment's pod template and set schedulerName to ack-co-scheduler:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-numa
  labels:
    app: nginx-numa
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-numa
  template:
    metadata:
      annotations:
        cpuset-scheduler: "true"   # Enable CPU topology-aware scheduling.
      labels:
        app: nginx-numa
    spec:
      schedulerName: ack-co-scheduler   # Route pods through the ACK co-scheduler.
      containers:
      - name: nginx-numa
        image: nginx:1.13.3
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 4
          limits:
            cpu: 4

Key fields:

Verify CPU topology-aware scheduling

After the deployment is running, confirm that pods were scheduled with cpuset pinning:

kubectl get pods -l app=nginx-numa -o wide

To confirm NUMA pinning on a specific node, log on to the node and check the cpuset assigned to the container:

cat /sys/fs/cgroup/cpuset/kubepods/pod<pod-uid>/<container-id>/cpuset.cpus

The output shows the CPU cores allocated to the container. If they all belong to the same NUMA node, cpuset pinning is active.

For more information, see Enable CPU topology-aware scheduling.

Prerequisites

Deploy the ack-virtual-node component before enabling this feature. See Use ECI in ACK.

Enable ECI elastic scheduling

Add the alibabacloud.com/burst-resource annotation to your Deployment's pod template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 4
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      name: nginx
      annotations:
        alibabacloud.com/burst-resource: eci   # Use ECI when ECS capacity is insufficient.
      labels:
        app: nginx
    spec:
      schedulerName: ack-co-scheduler   # Route pods through the ACK co-scheduler.
      containers:
      - name: nginx
        image: nginx
        resources:
          limits:
            cpu: 2
          requests:
            cpu: 2

Annotation values for `alibabacloud.com/burst-resource`:

Verify ECI elastic scheduling

After deploying, check which nodes the pods are running on:

kubectl get pods -l app=nginx -o wide

For more information, see Use ElasticResource to implement ECI elastic scheduling (deprecated).