Container Service for Kubernetes:Automate RayJob execution in an ACK cluster

ElasticQuotaTree defines the cluster's quota hierarchy. Each node specifies:

• min: guaranteed minimum resources for a team or department

• max: maximum resources it can use (borrowed from siblings if available)

When a job's guaranteed minimum cannot be met, the scheduler reclaims resources from quotas exceeding their guaranteed minimum — enabling preemption for high-priority jobs.

Only one ElasticQuotaTree is supported per cluster, and it must be in the kube-system namespace.

1. Apply the ElasticQuotaTree configuration. This example creates a three-level hierarchy: root → algorithm department → video team, with video bound to the video namespace.

   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     name: video
   
   ---
   apiVersion: scheduling.sigs.k8s.io/v1beta1
   kind: ElasticQuotaTree
   metadata:
     name: elasticquotatree        # Only one ElasticQuotaTree is supported per cluster
     namespace: kube-system        # Must be created in kube-system to take effect
   spec:
     root:
       name: root
       min:
         cpu: 100
         memory: 50Gi
         nvidia.com/gpu: 16
       max:
         cpu: 100
         memory: 50Gi
         nvidia.com/gpu: 16
       children:
       - name: algorithm
         min:
           cpu: 50
           memory: 25Gi
           nvidia.com/gpu: 10
         max:
           cpu: 80
           memory: 50Gi
           nvidia.com/gpu: 14
         children:
         - name: video
           min:
             cpu: 12
             memory: 12Gi
             nvidia.com/gpu: 2
           max:
             cpu: 14
             memory: 14Gi
             nvidia.com/gpu: 4
           namespaces:
           - video                 # Jobs in this namespace count against the video quota

   Jobs marked intern-text or intern-video have a guaranteed minimum of 0, so when an algorithm team member submits an urgent job while intern tasks are running, the scheduler can preempt the intern tasks' resources for the higher-priority job.

2. Verify the ElasticQuotaTree is active in kube-system.

   kubectl -n kube-system get elasticquotatree elasticquotatree -o yaml

Gang scheduling ensures all Pods in a distributed job schedule simultaneously. Without it, partial scheduling can leave a job waiting indefinitely.

Use gang scheduling when all nodes must start simultaneously:

• Large-scale ML training: multi-node training jobs where all workers must start together

• MPI jobs: master and worker processes that communicate from startup

• Streaming and analytics: jobs that process data in parallel across many nodes

• Custom distributed apps: game matchmaking services, IoT data collection pipelines

Add the ray.io/scheduler-name: kube-scheduler label to the RayJob metadata. The KubeRay Operator then injects gang scheduling labels onto every Pod it creates.

apiVersion: ray.io/v1
kind: RayJob
metadata:
  generateName: rayjob-sample-
  namespace: algorithm-text
  labels:
    ray.io/scheduler-name: kube-scheduler          # Enable gang scheduling
    quota.scheduling.alibabacloud.com/name: algorithm-video  # Bind to this quota
spec:
  entrypoint: python /home/ray/samples/sample_code.py
  runtimeEnvYAML: |
    pip:
      - requests==2.26.0
      - pendulum==2.1.2
    env_vars:
      counter_name: "test_counter"
  shutdownAfterJobFinishes: true
  suspend: true

  rayClusterSpec:
    rayVersion: '2.9.0'
    headGroupSpec:
      rayStartParams:
        dashboard-host: '0.0.0.0'
      template:
        spec:
          containers:
            - name: ray-head
              image: rayproject/ray:2.9.0
              ports:
                - containerPort: 6379
                  name: gcs-server
                - containerPort: 8265
                  name: dashboard
                - containerPort: 10001
                  name: client
              resources:
                limits:
                  cpu: "1"
                requests:
                  cpu: "1"
              volumeMounts:
                - mountPath: /home/ray/samples
                  name: code-sample
          volumes:
            - name: code-sample
              configMap:
                name: ray-job-code-sample
                items:
                  - key: sample_code.py
                    path: sample_code.py
    workerGroupSpecs:
      - replicas: 30
        groupName: small-group
        rayStartParams: {}
        template:
          spec:
            containers:
              - name: ray-worker
                image: rayproject/ray:2.9.0
                lifecycle:
                  preStop:
                    exec:
                      command: ["/bin/sh", "-c", "ray stop"]
                resources:
                  limits:
                    cpu: "1"
                  requests:
                    cpu: "1"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ray-job-code-sample
  namespace: algorithm-text
data:
  sample_code.py: |
    import ray
    import os
    import requests

    ray.init()

    @ray.remote
    class Counter:
        def __init__(self):
            self.name = os.getenv("counter_name")
            assert self.name == "test_counter"
            self.counter = 0

        def inc(self):
            self.counter += 1

        def get_counter(self):
            return "{} got {}".format(self.name, self.counter)

    counter = Counter.remote()

    for _ in range(5):
        ray.get(counter.inc.remote())
        print(ray.get(counter.get_counter.remote()))

    assert requests.__version__ == "2.26.0"

    import time
    time.sleep(30)

The KubeRay Operator injects these labels on each Pod to enable gang scheduling:

Label	Value	Purpose
pod-group.scheduling.sigs.k8s.io/min-available	31 (1 head + 30 workers)	Minimum Pods that must schedule together
pod-group.scheduling.sigs.k8s.io/name	RayCluster name	Groups Pods into the same scheduling unit
quota.scheduling.alibabacloud.com/name	algorithm-video	Binds Pods to the specified quota
scheduling.x-k8s.io/pod-group	RayCluster name	Pod group reference for the scheduler