How to manage AI/ML workloads with the ack-kube-queue job queue - Container Service for Kubernetes

How it works

ack-kube-queue uses Kubernetes' native suspend mechanism. When you submit a job with suspend set to true (via a field or annotation), the job enters the queue. ack-kube-queue monitors resource availability and sets suspend to false when resources are sufficient, starting the job.

Supported cluster types

ack-kube-queue supports:

ACK managed clusters running Kubernetes 1.18 or later
ACK Edge clusters running Kubernetes 1.18 or later
ACK Lingjun clusters running Kubernetes 1.18 or later

Install ack-kube-queue

Installation steps vary by cluster type.

ACK managed clusters and ACK Edge clusters

Choose the procedure for your cluster's current state.

If the cloud-native AI suite is not yet installed

Activate the cloud-native AI suite.
Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the target cluster name. In the left navigation pane, choose Applications > Cloud-native AI Suite.
At the bottom of the Cloud-native AI Suite page, click Deploy. In the Scheduling section, select Kube-Queue. In the Ecosystem Tools section, select Kubeflow and Arena. Then click Deploy Cloud-native AI Suite.

If the cloud-native AI suite is already installed

Activate the cloud-native AI suite.
Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the target cluster name. In the left navigation pane, choose Applications > Cloud-native AI Suite.
Install ack-arena and ack-kube-queue separately. Once installed, Status in Components shows Deployed.
- Find ack-arena and click Deploy in the Actions column. In the Parameters panel, click OK.
- Find ack-kube-queue and click Deploy in the Actions column. In the panel, click OK.

ACK Lingjun clusters

Log on to the ACK console. In the left navigation pane, choose Marketplace > Marketplace.
On the Marketplace page, search for ack-kube-queue and click its name.
On the application details page, click Deploy. In the Basic Information step, set Cluster, Namespace, and Release Name, then click Next.
In the Parameters step, set Chart Version to the latest version, then click OK.

Enable additional job types

By default, ack-kube-queue enables queuing only for Kubernetes-native jobs. To queue TensorFlow jobs, PyTorch jobs, MPI jobs, Argo workflows, Ray jobs, or Spark applications, enable each type individually.

Note

To queue Kubernetes-native jobs, the cluster must run Kubernetes 1.22 or later.

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the target cluster name. In the left navigation pane, choose Applications > Helm.
Find ack-kube-queue and click Update in the Actions column.

In the YAML template, set parameters for the job types to enable:

Parameter	Effect
`extension.argo.enable: true`	Enable Argo workflows
`extension.mpi.enable: true`	Enable MPI jobs
`extension.ray.enable: true`	Enable Ray jobs
`extension.spark.enable: true`	Enable Spark applications
`extension.tf.enable: true`	Enable TensorFlow jobs
`extension.pytorch.enable: true`	Enable PyTorch jobs

Click OK.

Submit jobs

Submit each supported job type to a queue and verify that it is queued.

Constraints

Job type	Constraint
TensorFlow, PyTorch, MPI	Must use the operator provided by ack-arena
MPI	Can only be submitted via Arena
Argo workflows	Only full workflows can be queued; individual workflow steps cannot. Declare resource requirements with the `kube-queue/min-resources` annotation (see Argo workflows).
Kubernetes-native jobs	Cluster must run Kubernetes 1.22 or later

TensorFlow jobs, PyTorch jobs, and MPI jobs

Add the annotation scheduling.x-k8s.io/suspend: "true" to the job manifest.

Example TensorFlow job:

apiVersion: "kubeflow.org/v1"
kind: "TFJob"
metadata:
  name: "job1"
  annotations:
    scheduling.x-k8s.io/suspend: "true"
spec:
...

Verify the job is queued:

kubectl describe tfjob job1

A queued job shows the Suspended condition.

Kubernetes-native jobs

Set spec.suspend to true. When dequeued, ack-kube-queue sets this field to false and the job starts.

apiVersion: batch/v1
kind: Job
metadata:
  generateName: pi-
spec:
  suspend: true
...

Verify the job is queued:

kubectl get job <job-name>

A queued job shows SUSPENDED. When ack-kube-queue admits the job, SUSPENDED clears and the job starts.

Argo workflows

Prerequisite: Install Argo Workflows from Marketplace in the ACK console.

Add a custom template named kube-queue-suspend of the suspend type, and set spec.suspend to true. Declare resource requirements with the kube-queue/min-resources annotation:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: <workflow-name>-
  annotations:
    kube-queue/min-resources: |
      cpu: 5
      memory: 5G
spec:
  suspend: true
  entrypoint: <entrypoint-template>
  templates:
  # Required: add a suspend template named kube-queue-suspend
  - name: kube-queue-suspend
    suspend: {}
  - name: <entrypoint-template>
    # ... your workflow steps

Spark applications

Prerequisite: Install ack-spark-operator from Marketplace in the ACK console.

Add the annotation scheduling.x-k8s.io/suspend: "true" to the SparkApplication manifest:

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  generateName: spark-pi-suspend-
  namespace: spark-operator
  annotations:
    scheduling.x-k8s.io/suspend: "true"
spec:
...

Ray jobs

Prerequisite: Install Kuberay-Operator from the Add-ons page in the ACK console. See Manage components.

Set spec.suspend to true:

apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: rayjob-sample
spec:
  # Suspend specifies whether the RayJob controller should create a RayCluster instance.
  # If a job is applied with the suspend field set to true, the RayCluster will not be created and we will wait for the transition to false.
  # If the RayCluster is already created, it will be deleted. In the case of transition to false, a new RayCluster will be created.
  suspend: true
...

Container Service for Kubernetes:Manage AI/ML workloads with ack-kube-queue