Container Service for Kubernetes：使用Arena提交TensorFlow單機訓練作業

通過以下程式碼範例提交 PS-Worker 模式下的 TensorFlow 分布式訓練作業，它包含 1 個 PS 節點，2 個 Worker 節點。

arena submit tf \
    --name=tf-mnist-dist \
    --namespace=default \
    --working-dir=/root \
    --ps=1 \
    --ps-image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/tensorflow-mnist-example:2.15.0-gpu \
    --workers=2 \
    --worker-image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/tensorflow-mnist-example:2.15.0-gpu \
    --gpus=1 \
    --sync-mode=git \
    --sync-source=https://github.com/kubeflow/arena.git \
    --env=GIT_SYNC_BRANCH=master \
    --data=training-data:/mnt \
    --tensorboard \
    --logdir=/mnt/tf_data/logs \
    "python /root/code/arena/examples/tensorflow/mnist/main.py --data /mnt/tf_data/mnist.npz --dir /mnt/tf_data/logs"

預期輸出：

service/tf-mnist-dist-tensorboard created
deployment.apps/tf-mnist-dist-tensorboard created
tfjob.kubeflow.org/tf-mnist-dist created
INFO[0004] The Job tf-mnist-dist has been submitted successfully
INFO[0004] You can run `arena get tf-mnist-dist --type tfjob -n default` to check the job status

NAME           ACCESSMODE     DESCRIPTION  OWNER  AGE
training-data  ReadWriteMany                      35m

 arena submit tf \
    --name=tf-mnist-dist \
    --namespace=default \
    --working-dir=/root \
    --ps=1 \
    --ps-image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/tensorflow-mnist-example:2.15.0-gpu \
    --workers=2 \
    --worker-image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/tensorflow-mnist-example:2.15.0-gpu \
    --gpus=1 \
    --sync-mode=git \
    --sync-source=https://github.com/kubeflow/arena.git \
    --env=GIT_SYNC_BRANCH=master \
    --env=GIT_SYNC_USERNAME=yourname \
    --env=GIT_SYNC_PASSWORD=yourpwd \
    --data=training-data:/mnt \
    --tensorboard \
    --logdir=/mnt/tf_data/logs \
    "python /root/code/arena/examples/tensorflow/mnist/main.py --data /mnt/tf_data/mnist.npz --dir /mnt/tf_data/logs"

arena submit tf \
    --name=tf-mnist-dist \
    --namespace=default \
    --working-dir=/root \
    --ps=1 \
    --ps-image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/tensorflow-mnist-example:2.15.0-gpu \
    --workers=2 \
    --worker-image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/tensorflow-mnist-example:2.15.0-gpu \
    --gpus=1 \
    --data=training-data:/mnt \
    --tensorboard \
    --logdir=/mnt/tf_data/logs \
    "python /code/github.com/kubeflow/arena/examples/tensorflow/mnist/main.py --data /mnt/tf_data/mnist.npz --dir /mnt/tf_data/logs"

arena logs -n default tf-dist

預期輸出：

WARNING:tensorflow:From code/tensorflow-sample-code/tfjob/docker/mnist/main.py:120: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:
...
Accuracy at step 960: 0.9691
Accuracy at step 970: 0.9677
Accuracy at step 980: 0.9687
Accuracy at step 990: 0.968
Adding run metadata for 999
Total Train-accuracy=0.968

使用上述命令擷取作業日誌資訊時，預設輸出 worker-0 節點的日誌。如果需要查看分布式訓練任務中的某個節點日誌，可以先查看作業詳情擷取作業的節點列表，然後使用命令arena logs $job_name -i $instance_name查看具體執行個體的日誌。

範例程式碼如下。

arena get tf-dist

預期輸出：

STATUS: SUCCEEDED
NAMESPACE: default
PRIORITY: N/A
TRAINING DURATION: 1m

NAME     STATUS     TRAINER  AGE  INSTANCE          NODE
tf-dist  SUCCEEDED  TFJOB    5m   tf-dist-ps-0      192.16x.x.xx
tf-dist  SUCCEEDED  TFJOB    5m   tf-dist-worker-0  192.16x.x.xx
tf-dist  SUCCEEDED  TFJOB    5m   tf-dist-worker-1  192.16x.x.xx

Your tensorboard will be available on:
http://192.16x.x.xx:31870

執行以下命令擷取作業日誌。

arena logs tf-dist -i tf-dist-worker-1

預期輸出：

WARNING:tensorflow:From code/tensorflow-sample-code/tfjob/docker/mnist/main.py:120: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:
...
Accuracy at step 970: 0.9676
Accuracy at step 980: 0.968
Accuracy at step 990: 0.967
Adding run metadata for 999
Total Train-accuracy=0.967

您還可以通過命令arena logs $job_name -f即時查看作業的日誌輸出，通過命令arena logs $job_name -t N查看尾部N行的日誌，以及通過arena logs --help查詢更多參數使用方式。

查看尾部N行的日誌範例程式碼如下。

arena logs tf-dist -t 5

預期輸出：

Accuracy at step 9970: 0.9834
Accuracy at step 9980: 0.9828
Accuracy at step 9990: 0.9816
Adding run metadata for 9999
Total Train-accuracy=0.9816