Container Service for Kubernetes:Best practices for monitoring cluster GPU resources

Usage notes

• GPU metrics are collected every 15 seconds, which may delay Grafana dashboard data. The dashboard might show no available GPU memory on a node while a Pod is still scheduled to it. This occurs when a Pod releases GPU resources between scrapes and the scheduler places a pending Pod before the next collection.

• The monitoring dashboard tracks only GPU resources requested via the resources.limits field in a Pod specification. See Manage Resources for Containers.

  Dashboard data may be inaccurate if you use GPU resources in any of the following ways:

  • Running GPU applications directly on the node.

  • Starting a container with docker run to run a GPU application.

  • Setting NVIDIA_VISIBLE_DEVICES, such as NVIDIA_VISIBLE_DEVICES=all or NVIDIA_VISIBLE_DEVICES=<GPU ID>, in the Pod's env section to request GPU resources.

  • Running a GPU program in a Pod with privileged: true in its securityContext.

  • Running a GPU program in a Pod without NVIDIA_VISIBLE_DEVICES set, where the container image defaults to NVIDIA_VISIBLE_DEVICES=all.

• Allocated GPU memory differs from used GPU memory. For example, a card has 16 GiB total and you allocate 5 GiB to a Pod. If the Pod runs sleep 1000, it is Running but does not use the GPU — allocated memory is 5 GiB, used memory is 0 GiB.

Step 1: Create node pools

The GPU monitoring dashboard displays metrics for Pods that request GPU resources by full card, GPU memory, or computing power.

Create three node pools to demonstrate Pod scheduling and resource usage for each GPU request mode. For instructions, see Create a node pool. Configure the node pools as follows:

Step 2: Deploy GPU applications

After you create the node pools, run GPU test jobs to verify that metrics are collected correctly. See Enable scheduling for label and scheduling details. The three jobs are as follows:

1. Create the Job manifest files.

   • Create a file named tensorflow-benchmark-exclusive.yaml.

     apiVersion: batch/v1
     kind: Job
     metadata:
       name: tensorflow-benchmark-exclusive
     spec:
       parallelism: 1
       template:
         metadata:
           labels:
             app: tensorflow-benchmark-exclusive
         spec:
           containers:
           - name: tensorflow-benchmark
             image: registry.cn-beijing.aliyuncs.com/ai-samples/gpushare-sample:benchmark-tensorflow-2.2.3
             command:
             - bash
             - run.sh
             - --num_batches=5000000
             - --batch_size=8
             resources:
               limits:
                 nvidia.com/gpu: 1 # Requests one GPU card.
             workingDir: /root
           restartPolicy: Never

   • Create a file named tensorflow-benchmark-share-mem.yaml.

     apiVersion: batch/v1
     kind: Job
     metadata:
       name: tensorflow-benchmark-share-mem
     spec:
       parallelism: 1
       template:
         metadata:
           labels:
             app: tensorflow-benchmark-share-mem
         spec:
           containers:
           - name: tensorflow-benchmark
             image: registry.cn-beijing.aliyuncs.com/ai-samples/gpushare-sample:benchmark-tensorflow-2.2.3
             command:
             - bash
             - run.sh
             - --num_batches=5000000
             - --batch_size=8
             resources:
               limits:
                 aliyun.com/gpu-mem: 10 # Requests 10 GiB of GPU memory.
             workingDir: /root
           restartPolicy: Never

   • Create a file named tensorflow-benchmark-share-mem-core.yaml.

     apiVersion: batch/v1
     kind: Job
     metadata:
       name: tensorflow-benchmark-share-mem-core
     spec:
       parallelism: 1
       template:
         metadata:
           labels:
             app: tensorflow-benchmark-share-mem-core
         spec:
           containers:
           - name: tensorflow-benchmark
             image: registry.cn-beijing.aliyuncs.com/ai-samples/gpushare-sample:benchmark-tensorflow-2.2.3
             command:
             - bash
             - run.sh
             - --num_batches=5000000
             - --batch_size=8
             resources:
               limits:
                 aliyun.com/gpu-mem: 10 # Requests 10 GiB of GPU memory.
                 aliyun.com/gpu-core.percentage: 30  # Requests 30% of the computing power of one card.
             workingDir: /root
           restartPolicy: Never

2. Deploy the Jobs:

   kubectl apply -f tensorflow-benchmark-exclusive.yaml
   kubectl apply -f tensorflow-benchmark-share-mem.yaml
   kubectl apply -f tensorflow-benchmark-share-mem-core.yaml

3. Check the Pod status:

   kubectl get pod

   Expected output:

   NAME                                        READY   STATUS    RESTARTS   AGE
   tensorflow-benchmark-exclusive-7dff2        1/1     Running   0          3m13s
   tensorflow-benchmark-share-mem-core-k24gz   1/1     Running   0          4m22s
   tensorflow-benchmark-share-mem-shmpj        1/1     Running   0          3m46s

   All Pods are in the Running state, confirming successful deployment.

Step 3: View the GPU dashboard