Container Service for Kubernetes:Schedule multi-cluster PyTorchJobs with gang scheduling and Kube Queue

Como funciona

Uma instância do Fleet agenda PyTorchJobs nos member clusters utilizando dois componentes:

• Kube Queue: gerencia as filas de jobs, aplica limites elásticos de cota e retém os jobs até que um member cluster tenha recursos suficientes.

• ACK Scheduler: aplica gang scheduling quando a instância do Fleet distribui pods para um member cluster e aloca todas as réplicas (Master e Workers) atomicamente.

Fluxo de agendamento:

1. Envie um PyTorchJob para a instância do Fleet com uma PropagationPolicy que defina customSchedulingType: Gang.

2. Se o gerenciamento de filas estiver ativado (suspension.scheduling: true), o Kube Queue reterá o job até haver um slot de cota disponível.

3. A instância do Fleet avalia os recursos dos member clusters e seleciona um cluster de destino.

4. O ACK Scheduler aloca todos os pods atomicamente no cluster selecionado.

5. A instância do Fleet monitora o job e sincroniza o status.

Pré-requisitos

Verifique se você tem:

• Conjunto de IA nativa da cloud instalado nos member clusters (apenas o componente Arena)

• Política AliyunAdcpFullAccess do RAM (Resource Access Management) anexada ao seu usuário RAM. Consulte Grant permissions to RAM users.

• Ferramenta de linha de comando AMC instalada. Consulte Use AMC.

• (Opcional) Reserva de recursos ativada para alinhar o agendamento do Fleet aos member clusters. Requer Kubernetes 1.28 ou posterior e ACK Scheduler 6.8.0 ou posterior.

Escolha um modo de agendamento

Dois modos de agendamento estão disponíveis:

Siga a Etapa 1 para configurar o gerenciamento de filas ou pule para a Etapa 2 caso deseje apenas o gang scheduling.

Etapa 1 (Opcional): Configurar filas de jobs com Kube Queue

Use ElasticQuotaTree para definir limites de cota e restringir jobs simultâneos entre namespaces.

1. Envie uma ElasticQuotaTree para a instância do Fleet. Este exemplo limita o namespace default a um job simultâneo, com até 10.000 CPUs, 10.000 GiB de memória e 10.000 GPUs.

   apiVersion: scheduling.sigs.k8s.io/v1beta1
   kind: ElasticQuotaTree
   metadata:
     name: elasticquotatree  # Only a single ElasticQuotaTree is supported.
     namespace: kube-system   # Must be created in the kube-system namespace.
   spec:
     root:
       name: root
       max:
         cpu: 999900
         memory: 400000Gi
         kube-queue/max-jobs: 10000000000
         nvidia.com/gpu: 100000
       min:
         cpu: 999900
         memory: 400000Gi
         kube-queue/max-jobs: 10000000000
         nvidia.com/gpu: 100000
       children:
       - name: child-2
         max:
           kube-queue/max-jobs: 1  # Only one job can be dequeued at a time.
           cpu: 10000
           nvidia.com/gpu: 10000
           memory: 10000Gi
         namespaces:
           - default

2. Verifique se o Kube Queue criou as filas:

   kubectl get queue -n kube-queue

   Saída esperada:

   NAME                 AGE
   root-child-2-v5zxz   15d
   root-kdzw7           15d

Etapa 2: Enviar um PyTorchJob para agendamento em múltiplos clusters

apiVersion: policy.one.alibabacloud.com/v1alpha1
kind: PropagationPolicy
metadata:
  name: example-policy
  namespace: default
spec:
  propagateDeps: true
  failover:
    application:
      decisionConditions:
        tolerationSeconds: 30
      purgeMode: Immediately
  placement:
    replicaScheduling:
      replicaSchedulingType: Divided
      customSchedulingType: Gang
  resourceSelectors:
    - apiVersion: kubeflow.org/v1
      kind: PyTorchJob

apiVersion: policy.one.alibabacloud.com/v1alpha1
kind: PropagationPolicy
metadata:
  name: example-policy
  namespace: default
spec:
  suspension:
    scheduling: true
  propagateDeps: true
  failover:
    application:
      decisionConditions:
        tolerationSeconds: 30
      purgeMode: Immediately
  placement:
    replicaScheduling:
      replicaSchedulingType: Divided
      customSchedulingType: Gang
  resourceSelectors:
    - apiVersion: kubeflow.org/v1
      kind: PyTorchJob

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  labels:
    app: pytorchjob
  name: pytorch-test
  namespace: default
spec:
  cleanPodPolicy: None
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      restartPolicy: Never
      template:
        metadata:
          labels:
            app: pytorchjob
          name: pytorch-test
        spec:
          schedulerName: default-scheduler
          containers:
          - command:
            - sh
            - -c
            - sleep 1h
            env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: void
            - name: gpus
              value: "0"
            - name: workers
              value: "8"
            image: registry-cn-hangzhou.ack.aliyuncs.com/acs/nginx
            imagePullPolicy: Always
            name: pytorch
            resources:
              limits:
                cpu: "3"
              requests:
                cpu: "10m"
            volumeMounts:
            - mountPath: /dev/shm
              name: dshm
            workingDir: /root
          volumes:
          - emptyDir:
              medium: Memory
              sizeLimit: 2Gi
            name: dshm
    Worker:
      replicas: 2
      restartPolicy: OnFailure
      template:
        metadata:
          labels:
            app: pytorchjob
          name: pytorch-test
        spec:
          containers:
          - command:
            - bash
            - -c
            - |
              echo "$WORKER_INDEX"
              sleep 1h
            env:
            - name: WORKER_INDEX
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['pytorch-replica-index']
            - name: NVIDIA_VISIBLE_DEVICES
              value: void
            - name: gpus
              value: "0"
            - name: workers
              value: "8"
            image: registry-cn-hangzhou.ack.aliyuncs.com/acs/nginx
            imagePullPolicy: Always
            name: pytorch
            resources:
              limits:
                cpu: "2"
              requests:
                cpu: "2"
                memory: "2Gi"
            volumeMounts:
            - mountPath: /dev/shm
              name: dshm
            workingDir: /root
          volumes:
          - emptyDir:
              medium: Memory
              sizeLimit: 2Gi
            name: dshm

Etapa 3: Verificar o status do job

Na instância do Fleet, execute os comandos abaixo para confirmar o agendamento do job e a execução de todos os pods.

1. Verifique o estado do PyTorchJob:

   kubectl get pytorchjob

   Saída esperada:

   NAME           STATE     AGE
   pytorch-test   Created   3m44s

2. Identifique o member cluster que recebeu o job:

   kubectl describe pytorchjob pytorch-test

   Nos eventos, procure por ScheduleBindingSucceed. O campo de resultado exibe o cluster de destino e a contagem de réplicas:

   Normal   ScheduleBindingSucceed  4m59s   default-scheduler   Binding has been scheduled successfully. Result: {cfxxxxxx:0,[{master 1} {worker 2}]}

   cfxxxxxx é o ID do member cluster onde todos os pods estão em execução.

3. Confirme a execução do job no member cluster:

   kubectl amc get pytorchjob -M

   Saída esperada:

   NAME           CLUSTER    STATE     AGE     ADOPTION
   pytorch-test   cfxxxxxx   Running   6m23s   Y

   ADOPTION: Y indica que a instância do Fleet assumiu o agendamento deste job.

4. Valide se todos os pods estão ativos:

   kubectl amc get pod -M

   Saída esperada:

   NAME                    CLUSTER    READY   STATUS    RESTARTS   AGE
   pytorch-test-master-0   cfxxxxxx   1/1     Running   0          7m16s
   pytorch-test-worker-0   cfxxxxxx   1/1     Running   0          7m16s
   pytorch-test-worker-1   cfxxxxxx   1/1     Running   0          7m16s

   Os três pods (um Master e dois Workers) executam no mesmo cluster, o que confirma o gang scheduling atômico.

5. Para inspecionar o YAML completo do PyTorchJob no member cluster:

   kubectl amc get pytorchjob pytorch-test -m ${member clusterid} -oyaml