Container Service for Kubernetes:Enable co-scheduling with ack-co-scheduler

Submit a TensorFlow distributed job

This example submits a TensorFlow distributed training job with Gang Scheduling. Both PS and Worker pods use pod-group.scheduling.sigs.k8s.io labels to form a pod group.

apiVersion: "kubeflow.org/v1"
kind: "TFJob"
metadata:
  name: "tf-smoke-gpu"
spec:
  tfReplicaSpecs:
    PS:
      replicas: 1
      template:
        metadata:
          creationTimestamp: null
          labels:
            pod-group.scheduling.sigs.k8s.io/name: tf-smoke-gpu
            pod-group.scheduling.sigs.k8s.io/min-available: "2"
        spec:
          schedulerName: ack-co-scheduler   # Route pods through the ACK co-scheduler.
          containers:
          - args:
            - python
            - tf_cnn_benchmarks.py
            - --batch_size=32
            - --model=resnet50
            - --variable_update=parameter_server
            - --flush_stdout=true
            - --num_gpus=1
            - --local_parameter_device=cpu
            - --device=cpu
            - --data_format=NHWC
            image: registry.cn-hangzhou.aliyuncs.com/kubeflow-images-public/tf-benchmarks-cpu:v20171202-bdab599-dirty-284af3
            name: tensorflow
            ports:
            - containerPort: 2222
              name: tfjob-port
            resources:
              limits:
                cpu: '10'
            workingDir: /opt/tf-benchmarks/scripts/tf_cnn_benchmarks
          restartPolicy: OnFailure
    Worker:
      replicas: 4
      template:
        metadata:
          creationTimestamp: null
          labels:
            pod-group.scheduling.sigs.k8s.io/name: tf-smoke-gpu
            pod-group.scheduling.sigs.k8s.io/min-available: "2"
        spec:
          schedulerName: ack-co-scheduler
          containers:
          - args:
            - python
            - tf_cnn_benchmarks.py
            - --batch_size=32
            - --model=resnet50
            - --variable_update=parameter_server
            - --flush_stdout=true
            - --num_gpus=1
            - --local_parameter_device=cpu
            - --device=gpu
            - --data_format=NHWC
            image: registry.cn-hangzhou.aliyuncs.com/kubeflow-images-public/tf-benchmarks-cpu:v20171202-bdab599-dirty-284af3
            name: tensorflow
            ports:
            - containerPort: 2222
              name: tfjob-port
            resources:
              limits:
                cpu: 10
            workingDir: /opt/tf-benchmarks/scripts/tf_cnn_benchmarks
          restartPolicy: OnFailure

Key fields:

Verify Gang scheduling

After submitting the job, verify that pods enter a pending state together:

kubectl get pods -l pod-group.scheduling.sigs.k8s.io/name=tf-smoke-gpu

Pods remain in Pending until at least min-available pods are simultaneously schedulable. This is expected. If pods stay pending, check Events for scheduling messages:

kubectl describe pod <pod-name>

See Use Gang scheduling.

Prerequisites

The resource-controller component is deployed. See Manage add-ons.

Enable CPU topology-aware scheduling

Add the cpuset-scheduler: "true" annotation to your Deployment's pod template and set schedulerName to ack-co-scheduler:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-numa
  labels:
    app: nginx-numa
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-numa
  template:
    metadata:
      annotations:
        cpuset-scheduler: "true"   # Enable CPU topology-aware scheduling.
      labels:
        app: nginx-numa
    spec:
      schedulerName: ack-co-scheduler   # Route pods through the ACK co-scheduler.
      containers:
      - name: nginx-numa
        image: nginx:1.13.3
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 4
          limits:
            cpu: 4

Key fields:

Verify CPU topology-aware scheduling

Once running, confirm pods were scheduled with cpuset pinning:

kubectl get pods -l app=nginx-numa -o wide

To confirm NUMA pinning, log on to the node and check the container's cpuset:

cat /sys/fs/cgroup/cpuset/kubepods/pod<pod-uid>/<container-id>/cpuset.cpus

If all listed cores belong to the same NUMA node, cpuset pinning is active.

See Enable CPU topology-aware scheduling.

Prerequisites

The ack-virtual-node component is deployed. See Use ECI in ACK.

Enable ECI elastic scheduling

Add the alibabacloud.com/burst-resource annotation to your Deployment's pod template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 4
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      name: nginx
      annotations:
        alibabacloud.com/burst-resource: eci   # Use ECI when ECS capacity is insufficient.
      labels:
        app: nginx
    spec:
      schedulerName: ack-co-scheduler   # Route pods through the ACK co-scheduler.
      containers:
      - name: nginx
        image: nginx
        resources:
          limits:
            cpu: 2
          requests:
            cpu: 2

Annotation values for `alibabacloud.com/burst-resource`:

Verify ECI elastic scheduling

After deploying, check which nodes the pods are running on:

kubectl get pods -l app=nginx -o wide

See Use ElasticResource to implement ECI elastic scheduling (deprecated).