Container Service for Kubernetes:Best practices for monitoring GPU resources

Important notes

• GPU monitoring metrics are collected at a 15-second interval, which can cause a slight data delay on the Grafana dashboard. As a result, the dashboard might show no available GPU memory on a Node, but a Pod is still successfully scheduled to it. This can happen if a Pod completes its task and releases GPU resources within a 15-second collection interval (between two scrapes), allowing the scheduler to place a pending Pod on that Node before the next metric update.

• The Monitoring Dashboard only monitors GPU resources requested through resources.limits in a Pod. For more information, see Resource Management for Pods and Containers.

  The data on the Monitoring Dashboard may be inaccurate if you use GPU resources in the following ways:

  • Run a GPU application directly on a Node.

  • Run a GPU application in a container started directly with the docker run command.

  • Request GPU resources for a Pod by setting the NVIDIA_VISIBLE_DEVICES=all or NVIDIA_VISIBLE_DEVICES=<GPU ID> environment variable directly in the Pod's env section and running a GPU program.

  • Configure privileged: true in a Pod's securityContext and run a GPU program.

  • Run a GPU program in a Pod where the NVIDIA_VISIBLE_DEVICES environment variable is not set, but the container image used by the Pod has NVIDIA_VISIBLE_DEVICES=all configured by default.

• The allocated GPU memory and used GPU memory are not always the same. For example, a GPU card has 16 GiB of total GPU memory. You allocate 5 GiB of it to a Pod whose startup command is sleep 1000. In this case, the Pod is in a Running state but will not use the GPU for 1000 seconds. As a result, 5 GiB of GPU memory is allocated, but the used GPU memory is 0 GiB.

Step 1: Create a node pool

The GPU Monitoring Dashboard displays metrics for Pods that request GPU resources either as a full card or by a specific amount of GPU memory, optionally with computing power.

This example creates three Node Pools in a cluster to demonstrate Pod scheduling and resource usage for different GPU request models. For detailed instructions on creating a Node Pool, see Create a node pool. The configurations for the Node Pools are as follows:

Step 2: Deploy GPU applications

After creating the Node Pools, run GPU test Jobs on the Nodes to verify that GPU metrics are collected correctly. For information about the labels and scheduling relationships required for each Job, see GPU node types and scheduling labels. The configurations for the three Jobs are as follows:

1. Create the Job manifest files.

   • Create a file named tensorflow-benchmark-exclusive.yaml with the following YAML content.

     apiVersion: batch/v1
     kind: Job
     metadata:
       name: tensorflow-benchmark-exclusive
     spec:
       parallelism: 1
       template:
         metadata:
           labels:
             app: tensorflow-benchmark-exclusive
         spec:
           containers:
           - name: tensorflow-benchmark
             image: registry.cn-beijing.aliyuncs.com/ai-samples/gpushare-sample:benchmark-tensorflow-2.2.3
             command:
             - bash
             - run.sh
             - --num_batches=5000000
             - --batch_size=8
             resources:
               limits:
                 nvidia.com/gpu: 1 #Apply for a GPU.
             workingDir: /root
           restartPolicy: Never

   • Create a file named tensorflow-benchmark-share-mem.yaml with the following YAML content.

     apiVersion: batch/v1
     kind: Job
     metadata:
       name: tensorflow-benchmark-share-mem
     spec:
       parallelism: 1
       template:
         metadata:
           labels:
             app: tensorflow-benchmark-share-mem
         spec:
           containers:
           - name: tensorflow-benchmark
             image: registry.cn-beijing.aliyuncs.com/ai-samples/gpushare-sample:benchmark-tensorflow-2.2.3
             command:
             - bash
             - run.sh
             - --num_batches=5000000
             - --batch_size=8
             resources:
               limits:
                 aliyun.com/gpu-mem: 10 #Apply for 10 GiB of GPU memory.
             workingDir: /root
           restartPolicy: Never

   • Create a file named tensorflow-benchmark-share-mem-core.yaml with the following YAML content.

     apiVersion: batch/v1
     kind: Job
     metadata:
       name: tensorflow-benchmark-share-mem-core
     spec:
       parallelism: 1
       template:
         metadata:
           labels:
             app: tensorflow-benchmark-share-mem-core
         spec:
           containers:
           - name: tensorflow-benchmark
             image: registry.cn-beijing.aliyuncs.com/ai-samples/gpushare-sample:benchmark-tensorflow-2.2.3
             command:
             - bash
             - run.sh
             - --num_batches=5000000
             - --batch_size=8
             resources:
               limits:
                 aliyun.com/gpu-mem: 10 #Apply for 10 GiB of GPU memory.
                 aliyun.com/gpu-core.percentage: 30  # Apply for 30% of the computing power of a GPU.
             workingDir: /root
           restartPolicy: Never

2. Run the following commands to deploy the Jobs.

   kubectl apply -f tensorflow-benchmark-exclusive.yaml
   kubectl apply -f tensorflow-benchmark-share-mem.yaml
   kubectl apply -f tensorflow-benchmark-share-mem-core.yaml

3. Run the following command to check the status of the Pods.

   kubectl get pod

   Expected output:

   NAME                                        READY   STATUS    RESTARTS   AGE
   tensorflow-benchmark-exclusive-7dff2        1/1     Running   0          3m13s
   tensorflow-benchmark-share-mem-core-k24gz   1/1     Running   0          4m22s
   tensorflow-benchmark-share-mem-shmpj        1/1     Running   0          3m46s

   The output shows that all Pods are in the Running state, which means the Jobs were deployed successfully.

Step 3: View the GPU monitoring dashboard