Container Service for Kubernetes:Submit a Ray job from the head node

Step 1: Find the head node Pod

List the Pods in your Ray cluster namespace:

kubectl get pod -n ${RAY_CLUSTER_NS}

Expected output:

NAME                                           READY   STATUS    RESTARTS   AGE
myfirst-ray-cluster-head-v7pbw                 2/2     Running   0          39m

Note the head node Pod name for the next step.

Step 2: Connect to the head node Pod

Open a Bash shell on the head node Pod. Replace myfirst-ray-cluster-head-v7pbw with your Pod name.

kubectl exec -it -n ${RAY_CLUSTER_NS} myfirst-ray-cluster-head-v7pbw -- bash

Step 3: Create the job script

Use echo or cat to save the following script as my_script.py:

import ray
import os

# Connect to a local or remote Ray cluster
ray.init()

# Define a remote actor that runs on 1 CPU
@ray.remote(num_cpus=1)
class Counter:
    def __init__(self):
        self.name = "test_counter"
        self.counter = 0

    def increment(self):
        self.counter += 1

    def get_counter(self):
        return "{} got {}".format(self.name, self.counter)

counter = Counter.remote()

# Run 10,000 increments across the cluster
for _ in range(10000):
    counter.increment.remote()
    print(ray.get(counter.get_counter.remote()))

Step 4: Run the job

python my_script.py

Expected output:

2024-01-24 04:25:27,286	INFO worker.py:1329 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS
2024-01-24 04:25:27,286	INFO worker.py:1458 -- Connecting to existing Ray cluster at address: 172.16.0.236:6379...
2024-01-24 04:25:27,295	INFO worker.py:1633 -- Connected to Ray cluster. View the dashboard at http://172.16.0.236:8265
test_counter got 0
test_counter got 1
test_counter got 2
test_counter got 3
...