Container Service for Kubernetes:Deploy and manage the Cloud-native AI Suite

Deploy the suite

1. Log in to the ACK console and click Clusters in the left navigation pane.

2. Click the cluster name, then choose Applications > Cloud-native AI Suite in the left pane.

3. Click Deploy.

4. Select components and click Deploy Cloud-native AI Suite. The system checks dependencies before deploying. After installation, the Components list lets you Deploy, Upgrade, or Uninstall individual components.

5. After installing ack-ai-dashboard and ack-ai-dev-console, links to AI Dashboard and AI Developer Console appear on the Cloud-native AI Suite page.

   

Configure AI Dashboard

Grant RAM permissions to the worker role

Before AI Dashboard can access cluster data, attach a custom RAM policy to the cluster's worker role.

1. Create a custom policy.

   1. Log in to the RAM console and choose Permissions > Policies in the left navigation pane.

   2. Click Create Policy, select the JSON tab, and add the following policy:

      {
          "Version": "1",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Action": [
                      "cs:*",
                      "log:GetProject",
                      "log:GetLogStore",
                      "log:GetConfig",
                      "log:GetMachineGroup",
                      "log:GetAppliedMachineGroups",
                      "log:GetAppliedConfigs",
                      "log:GetIndex",
                      "log:GetSavedSearch",
                      "log:GetDashboard",
                      "log:GetJob",
                      "ecs:DescribeInstances",
                      "ecs:DescribeSpotPriceHistory",
                      "ecs:DescribePrice",
                      "eci:DescribeContainerGroups",
                      "eci:DescribeContainerGroupPrice",
                      "log:GetLogStoreLogs",
                      "ims:CreateApplication",
                      "ims:UpdateApplication",
                      "ims:GetApplication",
                      "ims:ListApplications",
                      "ims:DeleteApplication",
                      "ims:CreateAppSecret",
                      "ims:GetAppSecret",
                      "ims:ListAppSecretIds",
                      "ims:ListUsers"
                  ],
                  "Resource": "*"
              }
          ]
      }

   3. Name the policy k8sWorkerRolePolicy-{ClusterID} and click OK.

2. Attach the policy to the cluster's worker role.

   1. In the RAM console, choose Identities > Roles and search for KubernetesWorkerRole-{ClusterID}.

   2. Click Grant Permission.

   3. In Select Policy, click Custom Policy, search for k8sWorkerRolePolicy-{ClusterID}, select it, and click OK.

Complete the AI Dashboard setup

1. On the Cloud-native AI Suite page, select Sample Console for Interaction Mode. A Note dialog box appears.

   • If Authorized is shown, skip to step 3.

   • If Unauthorized appears in red, complete the RAM permissions above and click Authorization Check. After authorization, Authorized is displayed.

2. Set Console Data Storage: Pre-installed MySQL for testing or ApsaraDB RDS for production. See Install and configure AI Dashboard and AI Developer Console.

3. Click Deploy Cloud-native AI Suite. AI Dashboard is ready when its status shows Ready.

(Optional) Create and accelerate a dataset

Mount datasets from Object Storage Service (OSS) as persistent volumes (PVs) and accelerate access in AI Dashboard.

Create a PV and PVC

1. Create a namespace:

   kubectl create ns demo-ns

2. Create a file named fashion-mnist.yaml:

   Placeholder	Description	Example
   fashion-mnist	OSS bucket name	my-dataset-bucket
   oss-cn-beijing.aliyuncs.com	OSS endpoint for the bucket's region	oss-cn-hangzhou.aliyuncs.com
   AKID	AccessKey ID	LTAI5tXxx
   AKSECRET	AccessKey secret	xXxXxXx

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: fashion-demo-pv
   spec:
     accessModes:
     - ReadWriteMany
     capacity:
       storage: 10Gi
     csi:
       driver: ossplugin.csi.alibabacloud.com
       volumeAttributes:
         bucket: fashion-mnist
         otherOpts: "-o max_stat_cache_size=0 -o allow_other"
         url: oss-cn-beijing.aliyuncs.com
         akId: "AKID"
         akSecret: "AKSECRET"
       volumeHandle: fashion-demo-pv
     persistentVolumeReclaimPolicy: Retain
     storageClassName: oss
     volumeMode: Filesystem
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: fashion-demo-pvc
     namespace: demo-ns
   spec:
     accessModes:
     - ReadWriteMany
     resources:
       requests:
         storage: 10Gi
     selector:
       matchLabels:
         alicloud-pvname: fashion-demo-pv
     storageClassName: oss
     volumeMode: Filesystem
     volumeName: fashion-demo-pv

   Replace these placeholders:

3. Apply the manifest:

   kubectl create -f fashion-mnist.yaml

4. Verify the PV and PVC are bound:

   kubectl get pv fashion-demo-pv
   kubectl get pvc fashion-demo-pvc -n demo-ns

   Expected output for the PV:

   NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS   AGE
   fashion-demo-pv   10Gi       RWX            Retain           Bound    demo-ns/fashion-demo-pvc     oss            8h

   Expected output for the PVC:

   NAME               STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   fashion-demo-pvc   Bound    fashion-demo-pv   10Gi       RWX            oss            8h

Accelerate the dataset

1. Log in to AI Dashboard as an administrator.

2. Choose Dataset > Dataset List in the left navigation pane.

3. Find the dataset (fashion-demo-pvc) and click Accelerate in the Actions columnOperator column.

   

Cluster dashboard

AI Dashboard opens the cluster dashboard by default, showing cluster-wide GPU health and allocation.

GPU Utilization shows whether the GPU ran any work during the sample window, not how efficiently. A node at 100% may run lightweight kernels rather than heavy parallel workloads. Pair with GPU Memory (Used/Total) for a fuller picture.

Node dashboard

Click Nodes in the upper-right corner of the Cluster page to view GPU metrics per node and per GPU device.

Training job dashboard

Click TrainingJobs in the upper-right corner of the Nodes page to view resource consumption and GPU efficiency per job.

Resource quota dashboard

Click Quota in the upper-right corner of the Training Jobs page to view quota consumption by resource type (CPU, memory, nvidia.com/gpu, aliyun.com/gpu-mem, aliyun.com/gpu).

Set up a quota tree

1. Create namespaces for each team. Existing namespaces must have no running pods before association with a quota node.

   kubectl create ns namespace1
   kubectl create ns namespace2
   kubectl create ns namespace3
   kubectl create ns namespace4

2. In AI Dashboard, create quota nodes and associate each leaf node with a namespace. Set Min and Max for each node.

Create users and user groups

A user can belong to multiple groups, and a group can contain multiple users. Associate users with groups to grant access to allocated resources.

1. Create a user. See Generate the kubeconfig file and logon token for a new user.

2. Create a user group. See Add a user group.

Capacity scheduling example

This example shows how the scheduler shares and reclaims CPU resources across four namespaces. The quota tree has this structure:

Quota configuration:

Walk-through:

Without elastic quotas, each leaf namespace uses only its Min (10 cores = 2 pods at 5 cores/pod). With elastic quotas and 40 cluster cores available, namespaces borrow idle capacity up to their Max.

Step 1: Deploy five pods to namespace1, each requesting 5 CPU cores (25 cores total).

With root.a.1 Max set to 20 cores, 4 pods run (20 cores). The fifth pod stays Pending.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx1
  namespace: namespace1
  labels:
    app: nginx1
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx1
  template:
    metadata:
      name: nginx1
      labels:
        app: nginx1
    spec:
      containers:
      - name: nginx1
        image: nginx
        resources:
          limits:
            cpu: 5
          requests:
            cpu: 5

Step 2: Deploy five pods to namespace2, each requesting 5 CPU cores.

With 20 cores remaining (40 - 20 from namespace1), 4 pods run. The fifth stays Pending. Both namespace1 and namespace2 now consume all 40 root cores.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx2
  namespace: namespace2
  labels:
    app: nginx2
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx2
  template:
    metadata:
      name: nginx2
      labels:
        app: nginx2
    spec:
      containers:
      - name: nginx2
        image: nginx
        resources:
          limits:
            cpu: 5
          requests:
            cpu: 5

Step 3: Deploy five pods to namespace3, each requesting 5 CPU cores.

No idle capacity remains. The scheduler reclaims 10 cores from root.a to guarantee root.b.1's minimum. It reclaims 5 cores from root.a.1 (reducing namespace1 from 4 running pods to 3) and 5 cores from root.a.2 (reducing namespace2 from 4 running pods to 3). With 10 reclaimed cores, 2 pods run in namespace3.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx3
  namespace: namespace3
  labels:
    app: nginx3
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx3
  template:
    metadata:
      name: nginx3
      labels:
        app: nginx3
    spec:
      containers:
      - name: nginx3
        image: nginx
        resources:
          limits:
            cpu: 5
          requests:
            cpu: 5

Step 4: Deploy five pods to namespace4, each requesting 5 CPU cores.

The scheduler reclaims another 10 cores from root.a to guarantee root.b.2's minimum: 5 cores from root.a.1 and 5 from root.a.2. After reclamation, namespace1 and namespace2 each have 2 running pods (10 cores), and namespace4 gets 2 running pods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx4
  namespace: namespace4
  labels:
    app: nginx4
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx4
  template:
    metadata:
      name: nginx4
      labels:
        app: nginx4
    spec:
      containers:
      - name: nginx4
        image: nginx
        resources:
          limits:
            cpu: 5
          requests:
            cpu: 5

Result:

Each team's minimum guarantee was honored. Borrowed capacity was reclaimed when other teams needed their minimums back.