Use the ack-co-scheduler component to enable co-scheduling - Container Service for Kubernetes

Compared with the native Kubernetes scheduler, the Container Service for Kubernetes (ACK) scheduler offers additional features such as Gang Scheduling, CPU topology-aware scheduling, and ECI elastic scheduling. This topic describes how to install the ack-co-scheduler component in a registered cluster to use ACK’s scheduling capabilities in your self-managed Kubernetes cluster. This lets you easily leverage ACK’s enhanced scheduling features for big data, AI, and other applications to improve application performance.

Prerequisites

Create a registered cluster and connect your self-managed Kubernetes cluster to it. For more information, see Create an ACK One registered cluster.

The system components must meet the version requirements listed in the following table.

Component	Version requirement
Kubernetes	1.18.8 or later
Helm	3.0 or later
Docker	19.03.5
Operating system	CentOS 7.6, CentOS 7.7, Ubuntu 16.04, Ubuntu 18.04, Alibaba Cloud Linux

Notes

When you deploy a Job, you must specify the name of the scheduler as ack-co-scheduler by setting .template.spec.schedulerName to ack-co-scheduler.

Install the ack-co-scheduler component

Install using onectl

Install onectl on your on-premises machine. For more information, see Use onectl to manage registered clusters.

Run the following command to install the ack-co-scheduler component.

onectl addon install ack-co-scheduler

Expected output:

Addon ack-co-scheduler, version **** installed.

Install using the console

Log on to the Container Service Management Console . In the navigation pane on the left, click Clusters.
On the Clusters page, click the name of your destination cluster. In the navigation pane on the left, click Add-ons.
On the Add-ons page, click the Others tab. Find the ack-co-scheduler component and click Install in the lower-right corner of the card.
In the confirmation dialog box, click OK.

Gang scheduling

ACK implements Gang scheduling based on the new Kube-scheduler framework to solve the all-or-nothing job scheduling problem that the native scheduler cannot support.

Use the following template to submit a TensorFlow distributed job to your cluster. For more information about running distributed TensorFlow jobs, see Use Gang scheduling.

apiVersion: "kubeflow.org/v1"
kind: "TFJob"
metadata:
  name: "tf-smoke-gpu"
spec:
  tfReplicaSpecs:
    PS:
      replicas: 1
      template:
        metadata:
          creationTimestamp: null
          labels:
            pod-group.scheduling.sigs.k8s.io/name: tf-smoke-gpu
            pod-group.scheduling.sigs.k8s.io/min-available: "2"
        spec:
          schedulerName: ack-co-scheduler   # Set the scheduler name to ack-co-scheduler.
          containers:
          - args:
            - python
            - tf_cnn_benchmarks.py
            - --batch_size=32
            - --model=resnet50
            - --variable_update=parameter_server
            - --flush_stdout=true
            - --num_gpus=1
            - --local_parameter_device=cpu
            - --device=cpu
            - --data_format=NHWC
            image: registry.cn-hangzhou.aliyuncs.com/kubeflow-images-public/tf-benchmarks-cpu:v20171202-bdab599-dirty-284af3
            name: tensorflow
            ports:
            - containerPort: 2222
              name: tfjob-port
            resources:
              limits:
                cpu: '10'
            workingDir: /opt/tf-benchmarks/scripts/tf_cnn_benchmarks
          restartPolicy: OnFailure
    Worker:
      replicas: 4
      template:
        metadata:
          creationTimestamp: null
          labels:
            pod-group.scheduling.sigs.k8s.io/name: tf-smoke-gpu
            pod-group.scheduling.sigs.k8s.io/min-available: "2"
        spec:
          schedulerName: ack-co-scheduler
          containers:
          - args:
            - python
            - tf_cnn_benchmarks.py
            - --batch_size=32
            - --model=resnet50
            - --variable_update=parameter_server
            - --flush_stdout=true
            - --num_gpus=1
            - --local_parameter_device=cpu
            - --device=gpu
            - --data_format=NHWC
            image: registry.cn-hangzhou.aliyuncs.com/kubeflow-images-public/tf-benchmarks-cpu:v20171202-bdab599-dirty-284af3
            name: tensorflow
            ports:
            - containerPort: 2222
              name: tfjob-port
            resources:
              limits:
                cpu: 10
            workingDir: /opt/tf-benchmarks/scripts/tf_cnn_benchmarks
          restartPolicy: OnFailure

CPU topology-aware scheduling

Before using CPU topology-aware scheduling, deploy the resource-controller component. For more information, see Manage add-ons.

Use the following template to enable CPU topology-aware scheduling in a deployment. For more information about CPU topology-aware scheduling, see Enable CPU topology-aware scheduling.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-numa
  labels:
    app: nginx-numa
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-numa
  template:
    metadata:
      annotations:
        cpuset-scheduler: "true"
      labels:
        app: nginx-numa
    spec:
      schedulerName: ack-co-scheduler # Set the scheduler name to ack-co-scheduler.
      containers:
      - name: nginx-numa
        image: nginx:1.13.3
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 4
          limits:
            cpu: 4

ECI elastic scheduling

ECI elastic scheduling is an elastic scheduling policy provided by Alibaba Cloud. When deploying a service, you can add annotations to specify using only ECS or ECI resources, or to automatically request ECI resources when ECS resources are insufficient. ECI elastic scheduling meets your varying elastic resource needs across different workload scenarios.

Before using ECI elastic scheduling, deploy the ack-virtual-node component in your cluster. For more information, see Use ECI in ACK.

Use the following template to enable ECI elastic scheduling in a deployment. For more information about using ECI elastic scheduling, see Use ElasticResource to implement ECI elastic scheduling (deprecated).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 4
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      name: nginx
      annotations:
        alibabacloud.com/burst-resource: eci # Add an annotation to specify the type of elastic resource.
      labels:
        app: nginx
    spec:
      schedulerName: ack-co-scheduler # Set the scheduler name to ack-co-scheduler.
      containers:
      - name: nginx
        image: nginx
        resources:
          limits:
            cpu: 2
          requests:
            cpu: 2

Configure the pod annotation alibabacloud.com/burst-resource under template.metadata to specify the type of elastic resource. The possible values for alibabacloud.com/burst-resource are as follows:

If you do not specify this annotation: Use only existing ECS resources in the cluster.
eci: Use ECI resources when ECS resources in the cluster are insufficient.
eci_only: Use only ECI resources and do not use ECS resources in the cluster.

Shared GPU scheduling

For more information about using shared GPU scheduling, see Run the shared GPU scheduling component, Monitor and isolate GPU resources, and Manage shared GPUs using node pools.