Multi-cluster PyTorchJob scheduling with priority queuing - Container Service for Kubernetes

PyTorch is a widely used machine learning framework for multi-machine, multi-GPU distributed training. In Kubernetes, you submit PyTorch workloads with PyTorchJob. This topic explains how to use ACK Kube Queue for job queuing and quota management on an ACK Fleet instance, and how to declare gang scheduling requirements when dispatching resources across member clusters.

Configure Kube Queue on an ACK Fleet instance to queue PyTorchJobs and apply gang scheduling so all pods land on the same member cluster atomically.

How it works

A Fleet instance schedules PyTorchJobs across member clusters with two components:

Kube Queue manages job queues, enforces elastic quota limits, and holds jobs until a member cluster has enough resources.
ACK Scheduler applies gang scheduling when the Fleet instance distributes pods to a member cluster, placing all replicas (Master and Workers) atomically.

Scheduling flow:

Submit a PyTorchJob to the Fleet instance with a PropagationPolicy that sets customSchedulingType: Gang.
If queue management is enabled (suspension.scheduling: true), Kube Queue holds the job until a quota slot is available.
The Fleet instance evaluates member-cluster resources and selects a target cluster.
ACK Scheduler places all pods atomically on the selected cluster.
The Fleet instance monitors the job and syncs status.

Prerequisites

Ensure you have:

Cloud-native AI suite installed on member clusters (Arena component only)
The AliyunAdcpFullAccess RAM (Resource Access Management) policy attached to your RAM user. See Grant permissions to RAM users.
The AMC command-line tool installed. See Use AMC.
(Optional) Resource reservation enabled to align Fleet scheduling with member clusters. Requires Kubernetes 1.28 or later and ACK Scheduler 6.8.0 or later.

(Optional) Enable resource reservation

Without resource reservation, the Fleet instance estimates capacity by summing remaining resources across member-cluster nodes. With reservation enabled, it holds actual capacity on the target cluster before committing, so Fleet scheduling matches the member cluster result.

Log on to the ACK console and click Clusters in the left navigation pane.
Click your cluster name. In the left navigation pane, click Add-ons.
On the Add-ons page, find Kube Scheduler and click Configuration.
In the Kube Scheduler Parameters dialog box, set enableReservation to true and click OK.

Choose a scheduling mode

Two scheduling modes are available:

Mode	When to use	Key configuration
Gang scheduling only	Atomic pod placement without queue management	Set `customSchedulingType: Gang` in the PropagationPolicy
Gang scheduling + queue management	Orderly queuing with quota enforcement when many jobs compete for limited resources	Set `customSchedulingType: Gang` and `suspension.scheduling: true`

Follow Step 1 for queue management, or skip to Step 2 for gang scheduling only.

Step 1 (Optional): Set up job queues with Kube Queue

Use ElasticQuotaTree to define quota limits and limit concurrent jobs across namespaces.

Submit an ElasticQuotaTree to the Fleet instance. This example limits the default namespace to one concurrent job, with up to 10,000 CPUs, 10,000 GiB of memory, and 10,000 GPUs.

apiVersion: scheduling.sigs.k8s.io/v1beta1
kind: ElasticQuotaTree
metadata:
  name: elasticquotatree  # Only a single ElasticQuotaTree is supported.
  namespace: kube-system   # Must be created in the kube-system namespace.
spec:
  root:
    name: root
    max:
      cpu: 999900
      memory: 400000Gi
      kube-queue/max-jobs: 10000000000
      nvidia.com/gpu: 100000
    min:
      cpu: 999900
      memory: 400000Gi
      kube-queue/max-jobs: 10000000000
      nvidia.com/gpu: 100000
    children:
    - name: child-2
      max:
        kube-queue/max-jobs: 1  # Only one job can be dequeued at a time.
        cpu: 10000
        nvidia.com/gpu: 10000
        memory: 10000Gi
      namespaces:
        - default

Verify that Kube Queue created the queues:

kubectl get queue -n kube-queue

Expected output:

NAME                 AGE
root-child-2-v5zxz   15d
root-kdzw7           15d

Step 2: Submit a PyTorchJob for multi-cluster scheduling

Submit a PropagationPolicy

A PropagationPolicy tells the Fleet instance how to distribute the PyTorchJob across member clusters and which scheduling mode to apply.

Gang scheduling only

Set customSchedulingType: Gang for atomic pod placement without queuing.

apiVersion: policy.one.alibabacloud.com/v1alpha1
kind: PropagationPolicy
metadata:
  name: example-policy
  namespace: default
spec:
  propagateDeps: true
  failover:
    application:
      decisionConditions:
        tolerationSeconds: 30
      purgeMode: Immediately
  placement:
    replicaScheduling:
      replicaSchedulingType: Divided
      customSchedulingType: Gang
  resourceSelectors:
    - apiVersion: kubeflow.org/v1
      kind: PyTorchJob

Gang scheduling with queue management

Add suspension.scheduling: true to hold the job in Kube Queue until a quota slot is available, then place all pods atomically.

apiVersion: policy.one.alibabacloud.com/v1alpha1
kind: PropagationPolicy
metadata:
  name: example-policy
  namespace: default
spec:
  suspension:
    scheduling: true
  propagateDeps: true
  failover:
    application:
      decisionConditions:
        tolerationSeconds: 30
      purgeMode: Immediately
  placement:
    replicaScheduling:
      replicaSchedulingType: Divided
      customSchedulingType: Gang
  resourceSelectors:
    - apiVersion: kubeflow.org/v1
      kind: PyTorchJob

Submit a PyTorchJob

Submit this PyTorchJob to the Fleet instance. It defines one Master pod and two Worker pods.

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  labels:
    app: pytorchjob
  name: pytorch-test
  namespace: default
spec:
  cleanPodPolicy: None
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      restartPolicy: Never
      template:
        metadata:
          labels:
            app: pytorchjob
          name: pytorch-test
        spec:
          schedulerName: default-scheduler
          containers:
          - command:
            - sh
            - -c
            - sleep 1h
            env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: void
            - name: gpus
              value: "0"
            - name: workers
              value: "8"
            image: registry-cn-hangzhou.ack.aliyuncs.com/acs/nginx
            imagePullPolicy: Always
            name: pytorch
            resources:
              limits:
                cpu: "3"
              requests:
                cpu: "10m"
            volumeMounts:
            - mountPath: /dev/shm
              name: dshm
            workingDir: /root
          volumes:
          - emptyDir:
              medium: Memory
              sizeLimit: 2Gi
            name: dshm
    Worker:
      replicas: 2
      restartPolicy: OnFailure
      template:
        metadata:
          labels:
            app: pytorchjob
          name: pytorch-test
        spec:
          containers:
          - command:
            - bash
            - -c
            - |
              echo "$WORKER_INDEX"
              sleep 1h
            env:
            - name: WORKER_INDEX
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['pytorch-replica-index']
            - name: NVIDIA_VISIBLE_DEVICES
              value: void
            - name: gpus
              value: "0"
            - name: workers
              value: "8"
            image: registry-cn-hangzhou.ack.aliyuncs.com/acs/nginx
            imagePullPolicy: Always
            name: pytorch
            resources:
              limits:
                cpu: "2"
              requests:
                cpu: "2"
                memory: "2Gi"
            volumeMounts:
            - mountPath: /dev/shm
              name: dshm
            workingDir: /root
          volumes:
          - emptyDir:
              medium: Memory
              sizeLimit: 2Gi
            name: dshm

Step 3: Verify the job status

On the Fleet instance, run these commands to confirm the job was scheduled and all pods are running.

Check the PyTorchJob state:

kubectl get pytorchjob

Expected output:

NAME           STATE     AGE
pytorch-test   Created   3m44s

Check which member cluster received the job:
```
kubectl describe pytorchjob pytorch-test
```
In the events, look for ScheduleBindingSucceed. The result field shows the target cluster and replica counts:
```
Normal   ScheduleBindingSucceed  4m59s   default-scheduler   Binding has been scheduled successfully. Result: {cfxxxxxx:0,[{master 1} {worker 2}]}
```
cfxxxxxx is the member cluster ID where all pods run.
Confirm the job is running on the member cluster:
```
kubectl amc get pytorchjob -M
```
Expected output:
```
NAME           CLUSTER    STATE     AGE     ADOPTION
pytorch-test   cfxxxxxx   Running   6m23s   Y
```
ADOPTION: Y means the Fleet instance has taken over scheduling for this job.

Confirm all pods are running:

kubectl amc get pod -M

Expected output:

NAME                    CLUSTER    READY   STATUS    RESTARTS   AGE
pytorch-test-master-0   cfxxxxxx   1/1     Running   0          7m16s
pytorch-test-worker-0   cfxxxxxx   1/1     Running   0          7m16s
pytorch-test-worker-1   cfxxxxxx   1/1     Running   0          7m16s

All three pods (one Master and two Workers) run on the same cluster, confirming atomic gang scheduling.

To inspect the full PyTorchJob YAML on the member cluster:

kubectl amc get pytorchjob pytorch-test -m ${member clusterid} -oyaml