Submit a standalone TensorFlow job using Arena - Container Service for Kubernetes

This topic shows how to submit a standalone TensorFlow training job with Arena and visualize it using TensorBoard.

Prerequisites

You have created a Kubernetes cluster with GPUs. For more information, see Create a Kubernetes cluster with GPUs.
The cluster nodes can access the public network. For more information, see Enable public network access for a cluster.
The Arena client is installed. For more information, see Configure the Arena client.
You have created a PVC instance named training-data, and the MNIST dataset is stored in the tf_data path. For more information, see Configure NAS shared storage.

Background

This example downloads source code from a Git repository and uses a dataset from a shared storage system based on NAS-managed PVs and PVCs. It assumes that you have a PVC instance named training-data (a shared storage volume) that contains the dataset in a directory named tf_data.

Procedure

Step 1: View GPU resources

arena top node

Expected output:

NAME                        IPADDRESS        ROLE    STATUS  GPU(Total)  GPU(Allocated)
cn-beijing.192.168.xxx.xxx  192.168.xxx.xxx  <none>  Ready   0           0
cn-beijing.192.168.xxx.xxx  192.168.xxx.xxx  <none>  Ready   0           0
cn-beijing.192.168.xxx.xxx  192.168.xxx.xxx  <none>  Ready   2           0
cn-beijing.192.168.xxx.xxx  192.168.xxx.xxx  <none>  Ready   2           0
---------------------------------------------------------------------------------------------------
Allocated/Total GPUs In Cluster:
0/4 (0.0%)

The output shows that the cluster has two GPU nodes. Each node has two idle GPU cards available for training jobs.

Step 2: Submit a TensorFlow job

Run the arena submit tfjob/tf [--flag] command to submit a TensorFlow job.

Use the following command to submit a standalone TensorFlow job that uses a single node and a single GPU card.

arena submit tf \
    --name=tf-mnist \
    --working-dir=/root \
    --workers=1 \
    --gpus=1 \
    --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/tensorflow-mnist-example:2.15.0-gpu \
    --sync-mode=git \
    --sync-source=https://github.com/kubeflow/arena.git \
    --env=GIT_SYNC_BRANCH=master \
    --data=training-data:/mnt \
    --tensorboard \
    --logdir=/mnt/tf_data/logs \
    "python /root/code/arena/examples/tensorflow/mnist/main.py --data /mnt/tf_data/mnist.npz --dir /mnt/tf_data/logs"

Expected output:

service/tf-mnist-tensorboard created
deployment.apps/tf-mnist-tensorboard created
tfjob.kubeflow.org/tf-mnist created
INFO[0005] The Job tf-mnist has been submitted successfully
INFO[0005] You can run `arena get tf-mnist --type tfjob -n default` to check the job status

The following table describes the parameters.

Parameter	Required	Description	Default
--name	Yes	The name of the job. It must be globally unique.	N/A
--working-dir	No	The directory where the command is run.	/root
--gpus	No	The number of GPU cards that the worker node of the job uses.	0
--image	Yes	The URL of the image for the training environment.	N/A
--sync-mode	No	The source code synchronization mode. Valid values are git and rsync. This example uses git.	N/A
--sync-source	No	The URL of the source code repository. This parameter is required when you specify --sync-mode. For git mode, this parameter can be the URL of any Git-based code repository, such as a GitHub project or an Alibaba Cloud Code project. The project code is downloaded to the code/ directory within the path specified by --working-dir. In this example, the path is /root/code/arena.	N/A
--data	No	Mounts a shared storage volume (PVC) into the job's container. The value format is pvc-name`:`mount-path. pvc-name is the name of an existing PVC. You can run `arena data list` to list available PVCs. mount-path is the destination path inside the container from which your training code reads data. Note Run `arena data list` to view the PVCs available in the cluster. `NAME ACCESSMODE DESCRIPTION OWNER AGE training-data ReadWriteMany 35m` If no PVC is available, you must create one. For more information, see Configure NAS shared storage.	N/A
--tensorboard	No	Enables a TensorBoard service for data visualization. Use this parameter with --logdir to specify the path to the event files that TensorBoard reads.	N/A
--logdir	No	Specifies the path to the event files for TensorBoard. This parameter is required when --tensorboard is specified.	/training_logs

Note

To use a private Git repository, set your credentials by using the GIT_SYNC_USERNAME and GIT_SYNC_PASSWORD environment variables.

arena submit tf \
    --name=tf-mnist \
    --working-dir=/root \
    --workers=1 \
    --gpus=1 \
    --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/tensorflow-mnist-example:2.15.0-gpu \
    --sync-mode=git \
    --sync-source=https://github.com/kubeflow/arena.git \
    --env=GIT_SYNC_BRANCH=master \
    --env=GIT_SYNC_USERNAME=yourname \
    --env=GIT_SYNC_PASSWORD=yourpwd \
    --data=training-data:/mnt \
    --tensorboard \
    --logdir=/mnt/tf_data/logs \
    "python /root/code/arena/examples/tensorflow/mnist/main.py --data /mnt/tf_data --dir /mnt/tf_data/logs"

Arena uses git-sync to synchronize source code, so you can use any environment variable supported by the git-sync project.

Important

If pulling the code fails due to network issues, you can download it manually to your shared storage system. Alternatively, the provided sample image includes the sample code at /code/github.com/kubeflow/arena/examples/tensorflow/mnist/main.py, which allows you to submit the job directly as shown below:

arena submit tf \
    --name=tf-mnist \
    --working-dir=/root \
    --workers=1 \
    --gpus=1 \
    --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/tensorflow-mnist-example:2.15.0-gpu \
    --data=training-data:/mnt \
    --tensorboard \
    --logdir=/mnt/tf_data/logs \
    "python /code/github.com/kubeflow/arena/examples/tensorflow/mnist/main.py --data /mnt/tf_data/mnist.npz --dir /mnt/tf_data/logs"

Step 3: Check job status

Run the following command to list all jobs submitted using Arena.

arena list

Expected output:

NAME      STATUS   TRAINER  DURATION  GPU(Requested)  GPU(Allocated)  NODE
tf-mnist  RUNNING  TFJOB    3s        1               1               192.168.xxx.xxx

Run the following command to check the GPU resources used by the job.

arena top job

Expected output:

NAME      STATUS   TRAINER  AGE  GPU(Requested)  GPU(Allocated)  NODE
tf-mnist  RUNNING  TFJOB    29s  1               1               192.168.xxx.xxx
Total Allocated/Requested GPUs of Training Jobs: 1/1

Run the following command to check the GPU resources used by the cluster.

arena top node

Expected output:

NAME                        IPADDRESS        ROLE    STATUS  GPU(Total)  GPU(Allocated)
cn-beijing.192.168.xxx.xxx  192.168.xxx.xxx  <none>  Ready   0           0
cn-beijing.192.168.xxx.xxx  192.168.xxx.xxx  <none>  Ready   0           0
cn-beijing.192.168.xxx.xxx  192.168.xxx.xxx  <none>  Ready   2           1
cn-beijing.192.168.xxx.xxx  192.168.xxx.xxx  <none>  Ready   2           0
---------------------------------------------------------------------------------------------------
Allocated/Total GPUs In Cluster:
1/4 (25.0%)

Run the following command to view the details of the training job.

arena get -n default tf-mnist

Expected output:

Name:        tf-mnist
Status:      RUNNING
Namespace:   default
Priority:    N/A
Trainer:     TFJOB
Duration:    22s
CreateTime:  2026-01-26 16:01:42
EndTime:
Instances:
  NAME              STATUS   AGE  IS_CHIEF  GPU(Requested)  NODE
  ----              ------   ---  --------  --------------  ----
  tf-mnist-chief-0  Running  45s  true      1               cn-beijing.192.168.xxx.xxx
Tensorboard:
  Your tensorboard will be available on:
  http://192.168.xxx.xxx:31243

Note

Because TensorBoard is enabled, the output includes the TensorBoard URL. This URL is not displayed if TensorBoard is disabled.

Step 4: View TensorBoard

Use a web browser to view the TensorBoard dashboard.

Run the following command to forward the TensorBoard service in the cluster to port 9090 on your local machine.

Important
Port forwarding with kubectl port-forward is not suitable for production because it lacks reliability, security, and scalability. Use it only for development and debugging. For production-grade networking solutions in Kubernetes clusters, see Ingress management.
```
kubectl port-forward -n default svc/tf-mnist-tensorboard 9090:6006
```
Open http://localhost:9090 in your browser to view the TensorBoard dashboard.

The SCALARS panel in TensorBoard displays two curves: train (red) and test (blue). The accuracy_1 metric converges to over 0.96 after about 1,000 steps, and the cross_entropy_1 metric drops to about 0.05 to 0.10. This indicates that the MNIST model trained successfully. In the left-side panel, you can adjust display options such as Smoothing (currently 0.6) and Horizontal Axis (currently STEP). The log path is /mnt/tf_data/logs.

Step 5: View job logs

Run the following command to get the job logs.

arena logs -n default tf-mnist

Expected output:

Train Epoch: 14 [55680/60000 (93%)]     Loss: 0.029811
Train Epoch: 14 [56320/60000 (94%)]     Loss: 0.029721
Train Epoch: 14 [56960/60000 (95%)]     Loss: 0.029682
Train Epoch: 14 [57600/60000 (96%)]     Loss: 0.029781
Train Epoch: 14 [58240/60000 (97%)]     Loss: 0.029708
Train Epoch: 14 [58880/60000 (98%)]     Loss: 0.029761
Train Epoch: 14 [59520/60000 (99%)]     Loss: 0.029684
Test Accuracy: 9842/10000 (98.42%)
938/938 - 3s - loss: 0.0299 - accuracy: 0.9924 - val_loss: 0.0446 - val_accuracy: 0.9842 - lr: 0.0068 - 3s/epoch - 3ms/step

Note

To stream the job logs in real time, add the -f parameter.
To view only the last N lines of the logs, add the -t N or --tail N parameter.
For more options, run arena logs --help.

(Optional) Step 6: Clean up environment

After the training job is complete, you can delete it by running the following command:

arena delete -n default tf-mnist

Expected output:

INFO[0002] The training job tf-mnist has been deleted successfully