Container Service for Kubernetes:Best practices for using Arena in a multi-user scenario

Resource allocation

department1-public-dev1 and department1-public-dev2 are mounted to the same directory on the Apsara File Storage NAS (NAS) file system, so users in both groups can access that data. dev1-public and dev2-public are mounted to separate directories and are accessible only to Bob and Tom respectively.

Example cluster

All operations in this topic are performed using an administrator account on the client, unless otherwise stated.

Create Linux users and groups

Create user IDs (UIDs) and group IDs (GIDs) for Bob and Tom on the client. The Linux account system enforces Task 2: each user can only log in with their own credentials and runs Arena in their own environment.

# Create Linux groups: dev1 and dev2.
groupadd -g 10001 dev1
groupadd -g 10002 dev2

# Create Linux users: Bob and Tom.
adduser -u 20001 -s /bin/bash -G dev1 -m bob
adduser -u 20002 -s /bin/bash -G dev2 -m tom

# Set passwords.
passwd bob
passwd tom

Create Kubernetes namespaces and service accounts

After jobs are submitted, they run in the ACK cluster. Each Linux user group maps to a Kubernetes namespace, and each Linux user maps to a service account within that namespace.

Log in to the client as the root user and run the following commands. For prerequisites on connecting kubectl to the ACK cluster, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

kubectl version 1.10 or later is required.

# Create namespaces for dev1 and dev2.
kubectl create namespace dev1
kubectl create namespace dev2

# Create service accounts for Bob and Tom.
kubectl create serviceaccount bob -n dev1
kubectl create serviceaccount tom -n dev2

Expected output:

namespace/dev1 created
namespace/dev2 created
serviceaccount/bob created
serviceaccount/tom created

Install Arena

Install Arena once on the client. Log in as the root user, download the latest community release package, decompress it, and run install.sh. For more information, see Configure the Arena client.

Generate kubeconfig files

Each user needs their own kubeconfig file to access the ACK cluster with a scoped identity. Create a script named generate-kubeconfig.sh with the following content:

#!/usr/bin/env bash

set -e

NAMESPACE=
SERVICE_ACCOUNT=
DURATION=
OUTPUT=

help() {
    echo "Usage: $0 -n <namespace> -s <service-account> -d <duration> -o <output-file>"
    echo ""
    echo "Options:"
    echo "-n, --namespace <namespace>      Namespace of the service account."
    echo "-s, --service-account <name>     Name of the service account."
    echo "-d, --duration <duration>        Duration of the token e.g. 30d."
    echo "-o, --output <file>              Output file name. If not set, a temporary file will be created."
}

parse() {
    while [ $# -gt 0 ]; do
        case $1 in
        -n | --namespace)
            NAMESPACE="$2"
            shift 2
            ;;
        -s | --service-account)
            SERVICE_ACCOUNT="$2"
            shift 2
            ;;
        -d | --duration)
            DURATION="$2"
            shift 2
            ;;
        -o | --output)
            OUTPUT="$2"
            shift 2
            ;;
        *)
            help
            exit 0
            ;;
        esac
    done

    if [ -z "${NAMESPACE}" ] || [ -z "${SERVICE_ACCOUNT}" ] || [ -z "${DURATION}" ]; then
        help
        exit 0
    fi

    if [ -z "${OUTPUT}" ]; then
        OUTPUT=$(mktemp -d)/config
    elif [ -f "${OUTPUT}" ]; then
        echo "Output file \"${OUTPUT}\" already exists."
        exit 1
    fi
}

# Generate kubeconfig
generate_kubeconfig() {
    CONTEXT=$(kubectl config current-context)
    CLUSTER=$(kubectl config view -o jsonpath="{.contexts[?(@.name==\"${CONTEXT}\")].context.cluster}")
    SERVER=$(kubectl config view -o jsonpath="{.clusters[?(@.name==\"${CLUSTER}\")].cluster.server}")
    TOKEN=$(kubectl create token "${SERVICE_ACCOUNT}" --namespace "${NAMESPACE}" --duration="${DURATION}")
    CERT=$(mktemp)

    mkdir -p "$(dirname "${OUTPUT}")"
    kubectl config view --raw=true -o jsonpath="{.clusters[?(@.name==\"${CLUSTER}\")].cluster.certificate-authority-data}" | base64 -d >"${CERT}"
    kubectl config set-cluster "${CLUSTER}" --kubeconfig="${OUTPUT}" --server="${SERVER}" --embed-certs=true --certificate-authority="${CERT}" >/dev/null
    kubectl config set-credentials "${SERVICE_ACCOUNT}" --kubeconfig="${OUTPUT}" --token="${TOKEN}" >/dev/null
    kubectl config set-context "${CLUSTER}-${NAMESPACE}-${SERVICE_ACCOUNT}-context" --kubeconfig="${OUTPUT}" --cluster="${CLUSTER}" --user="${SERVICE_ACCOUNT}" --namespace="${NAMESPACE}" >/dev/null
    kubectl config use-context "${CLUSTER}-${NAMESPACE}-${SERVICE_ACCOUNT}-context" --kubeconfig="${OUTPUT}" >/dev/null
    rm "${CERT}"
    echo "Saved kubeconfig to \"${OUTPUT}\"."
}

main() {
    parse "$@"
    generate_kubeconfig
}

main "$@"

Generate kubeconfig files for Bob and Tom. The following commands set the token expiration to 720 hours — adjust this value based on your security policy.

bash generate-kubeconfig.sh -n dev1 -s bob -d 720h -o /home/bob/.kube/config
bash generate-kubeconfig.sh -n dev2 -s tom -d 720h -o /home/tom/.kube/config

Expected output:

Saved kubeconfig to "/home/bob/.kube/config".
Saved kubeconfig to "/home/tom/.kube/config".

After this step, Tasks 1 and 2 are complete. Bob and Tom can each log in to the client and run Arena in their own isolated environment.

Create roles

Create roles in the dev1 and dev2 namespaces. Bob and Tom are granted the minimum permissions needed to view and manage only their own jobs (Task 3).

The following table summarizes the permissions granted to each user:

Create dev1_roles.yaml for user group dev1:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: arena-topnode
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - deployments
  - nodes
  - nodes/*
  - services/proxy
  - persistentvolumes
  verbs:
  - get
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: arena
  namespace: dev1
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - '*'
  - apiGroups:
      - ""
    resources:
      - services/proxy
      - persistentvolumeclaims
      - events
    verbs:
      - get
      - list
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/log
      - services
    verbs:
      - '*'
  - apiGroups:
      - ""
      - apps
      - extensions
    resources:
      - deployments
      - replicasets
    verbs:
      - '*'
  - apiGroups:
      - kubeflow.org
    resources:
      - '*'
    verbs:
      - '*'
  - apiGroups:
      - batch
    resources:
      - jobs
    verbs:
      - '*'

Create dev2_roles.yaml for user group dev2:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: arena-topnode
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - deployments
  - nodes
  - nodes/*
  - services/proxy
  - persistentvolumes
  verbs:
  - get
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: arena
  namespace: dev2
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - services/proxy
  - persistentvolumeclaims
  - events
  verbs:
  - get
  - list
- apiGroups:
  - ""
  resources:
  - pods
  - pods/log
  - services
  verbs:
  - '*'
- apiGroups:
  - ""
  - apps
  - extensions
  resources:
  - deployments
  - replicasets
  verbs:
  - '*'
- apiGroups:
  - kubeflow.org
  resources:
  - '*'
  verbs:
  - '*'
- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - '*'

Apply both role definition files:

kubectl apply -f dev1_roles.yaml
kubectl apply -f dev2_roles.yaml

Expected output:

clusterrole.rbac.authorization.k8s.io/arena-topnode created
role.rbac.authorization.k8s.io/arena created

clusterrole.rbac.authorization.k8s.io/arena-topnode unchanged
role.rbac.authorization.k8s.io/arena created

Verify the roles were created:

kubectl get role -n dev1
kubectl get role -n dev2

Expected output:

NAME    CREATED AT
arena   2024-09-14T08:25:34Z

NAME    CREATED AT
arena   2024-09-14T08:25:39Z

Bind roles to users

Create bob_rolebindings.yaml:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: bob-arena-topnode
  namespace: dev1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: arena-topnode
subjects:
- kind: ServiceAccount
  name: bob
  namespace: dev1
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: bob-arena
  namespace: dev1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: arena
subjects:
- kind: ServiceAccount
  name: bob
  namespace: dev1

Create tom_rolebindings.yaml:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: tom-arena-topnode
  namespace: dev2
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: arena-topnode
subjects:
- kind: ServiceAccount
  name: tom
  namespace: dev2
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: tom-arena
  namespace: dev2
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: arena
subjects:
- kind: ServiceAccount
  name: tom
  namespace: dev2

Apply both files:

kubectl apply -f bob_rolebindings.yaml
kubectl apply -f tom_rolebindings.yaml

Expected output:

clusterrolebinding.rbac.authorization.k8s.io/bob-arena-topnode created
rolebinding.rbac.authorization.k8s.io/bob-arena created

clusterrolebinding.rbac.authorization.k8s.io/tom-arena-topnode created
rolebinding.rbac.authorization.k8s.io/tom-arena created

Verify the role bindings:

kubectl get rolebinding -n dev1
kubectl get rolebinding -n dev2

Expected output:

NAME        ROLE         AGE
bob-arena   Role/arena   34s

NAME        ROLE         AGE
tom-arena   Role/arena   33s

Tasks 1–3 are now complete.

Create a NAS file system

Log in to the NAS console, create a NAS file system, and add a mount target. For more information, see Configure a shared NAS volume.Alibaba Cloud NAS console

Create persistent volumes and claims

1. Create four persistent volumes (PVs) — dev1-public, dev2-public, department1-public-dev1, and department1-public-dev2. Mount department1-public-dev1 and department1-public-dev2 to the same NAS directory so both groups can access the shared data. Mount dev1-public and dev2-public to separate directories. For more information, see Mount a statically provisioned NAS volume.

   Select the mount target you added in the previous step.

   

2. Create one persistent volume claim (PVC) for each PV. For more information, see Mount a statically provisioned NAS volume. After the PVCs are created, department1-public-dev1 and dev1-public appear in namespace dev1, and department1-public-dev2 and dev2-public appear in namespace dev2.

Verify volume configuration

Log in to the client as the root user and run:

# Query volumes available to dev1.
arena data list -n dev1

# Query volumes available to dev2.
arena data list -n dev2

Expected output:

All five tasks are now complete.

Bob's account

1. Log in to the client as Bob and query available volumes:

   ssh bob@39.98.xxx.xx
   arena data list

   Expected output:

   

2. Submit a training job that requires one GPU:

   arena submit tf \
           --name=tf-git-bob-01 \
           --gpus=1 \
           --image=tensorflow/tensorflow:1.5.0-devel-gpu \
           --sync-mode=git \
           --sync-source=https://code.aliyun.com/xiaozhou/tensorflow-sample-code.git \
           "python code/tensorflow-sample-code/tfjob/docker/mnist/main.py --max_steps 10000 --data_dir=code/tensorflow-sample-code/data"

3. List all jobs submitted by Bob:

   arena list

   Expected output:

   

4. Submit a second job requesting one GPU:

   arena submit tf \
           --name=tf-git-bob-02 \
           --gpus=1 \
           --image=tensorflow/tensorflow:1.5.0-devel-gpu \
           --sync-mode=git \
           --sync-source=https://code.aliyun.com/xiaozhou/tensorflow-sample-code.git \
           "python code/tensorflow-sample-code/tfjob/docker/mnist/main.py --max_steps 10000 --data_dir=code/tensorflow-sample-code/data"

   dev1 has a quota of only 1 GPU, which the first job already occupies. The ACK cluster suspends the second job even though the cluster overall still has free GPUs. To confirm the quota is exhausted, run:

   kubectl describe resourcequotas dev1-compute-resources -n dev1

Tom's account

1. Log in to the client as Tom and query available volumes:

   ssh tom@39.98.xx.xx
   arena data list

   Expected output:

   

2. List all jobs submitted by Tom:

   arena list

   Tom cannot see the jobs submitted by Bob — namespace isolation prevents cross-group job visibility.

   

3. Submit a training job. Because dev2 has CPU and memory quotas, specify both when submitting:

    arena submit tf \
            --name=tf-git-tom-01 \
            --gpus=1 \
            --chief-cpu=2 \
            --chief-memory=10Gi \
            --image=tensorflow/tensorflow:1.5.0-devel-gpu \
            --sync-mode=git \
            --sync-source=https://code.aliyun.com/xiaozhou/tensorflow-sample-code.git \
            "python code/tensorflow-sample-code/tfjob/docker/mnist/main.py --max_steps 10000 --data_dir=code/tensorflow-sample-code/data"

4. Submit a second job:

    arena submit tf \
            --name=tf-git-tom-02 \
            --gpus=1 \
            --chief-cpu=2 \
            --chief-memory=10Gi \
            --image=tensorflow/tensorflow:1.5.0-devel-gpu \
            --sync-mode=git \
            --sync-source=https://code.aliyun.com/xiaozhou/tensorflow-sample-code.git \
            "python code/tensorflow-sample-code/tfjob/docker/mnist/main.py --max_steps 10000 --data_dir=code/tensorflow-sample-code/data"

5. List all jobs submitted by Tom:

   arena list

   Expected output: Tom's dev2 group has a quota of 2 GPUs, so both jobs run.