Container Service for Kubernetes:Best practices for monitoring cluster GPU resources

Usage notes

• GPU metrics are collected at a 15-second interval. This may cause a delay in the data displayed on the Grafana monitoring dashboards. As a result, the dashboard might show no available GPU memory on a node, yet a Pod is still successfully scheduled to it. This can happen if a running Pod releases its GPU resources between scrapes (within a 15-second window), and the scheduler places a pending Pod on the node before the next metric collection.

• The monitoring dashboard tracks only GPU resources requested by configuring the resources.limits field in a Pod specification. For more information, see Manage Resources for Containers.

  The monitoring dashboard data may be inaccurate if you use GPU resources in any of the following ways:

  • Running GPU applications directly on the node.

  • Starting a container to run a GPU application by using the docker run command.

  • Requesting GPU resources for a Pod by setting the NVIDIA_VISIBLE_DEVICES environment variable, such as NVIDIA_VISIBLE_DEVICES=all or NVIDIA_VISIBLE_DEVICES=<GPU ID>, in the Pod's env section and then running a GPU program.

  • Running a GPU program in a Pod with privileged: true configured in its securityContext.

  • Running a GPU program in a Pod that does not have the NVIDIA_VISIBLE_DEVICES environment variable set, but the Pod's container image sets NVIDIA_VISIBLE_DEVICES=all by default.

• The allocated GPU memory for a GPU card is not always the same as the used GPU memory. For example, a card has 16 GiB of total GPU memory, and you allocate 5 GiB of it to a Pod. If the Pod's startup command is sleep 1000, the Pod is in a Running state but will not use the GPU for 1,000 seconds. In this case, the allocated GPU memory is 5 GiB, but the used GPU memory is 0 GiB.

Step 1: Create node pools

The GPU monitoring dashboard can display metrics for Pods that request GPU resources by Full card or by GPU memory, including requests for computing power.

In this example, you will create three node pools in the cluster to demonstrate Pod scheduling and resource usage for different GPU request modes. For detailed instructions on how to create a node pool, see Create a node pool. The configurations for the node pools are as follows:

Step 2: Deploy GPU applications

After you create the node pools, run GPU test jobs on the nodes to verify that GPU metrics are collected correctly. For more information about the labels and scheduling relationships for each job, see Enable scheduling. The details of the three jobs are as follows:

1. Create the Job manifest files.

   • Create a file named tensorflow-benchmark-exclusive.yaml with the following YAML content.

     apiVersion: batch/v1
     kind: Job
     metadata:
       name: tensorflow-benchmark-exclusive
     spec:
       parallelism: 1
       template:
         metadata:
           labels:
             app: tensorflow-benchmark-exclusive
         spec:
           containers:
           - name: tensorflow-benchmark
             image: registry.cn-beijing.aliyuncs.com/ai-samples/gpushare-sample:benchmark-tensorflow-2.2.3
             command:
             - bash
             - run.sh
             - --num_batches=5000000
             - --batch_size=8
             resources:
               limits:
                 nvidia.com/gpu: 1 # Requests one GPU card.
             workingDir: /root
           restartPolicy: Never

   • Create a file named tensorflow-benchmark-share-mem.yaml with the following YAML content.

     apiVersion: batch/v1
     kind: Job
     metadata:
       name: tensorflow-benchmark-share-mem
     spec:
       parallelism: 1
       template:
         metadata:
           labels:
             app: tensorflow-benchmark-share-mem
         spec:
           containers:
           - name: tensorflow-benchmark
             image: registry.cn-beijing.aliyuncs.com/ai-samples/gpushare-sample:benchmark-tensorflow-2.2.3
             command:
             - bash
             - run.sh
             - --num_batches=5000000
             - --batch_size=8
             resources:
               limits:
                 aliyun.com/gpu-mem: 10 # Requests 10 GiB of GPU memory.
             workingDir: /root
           restartPolicy: Never

   • Create a file named tensorflow-benchmark-share-mem-core.yaml with the following YAML content.

     apiVersion: batch/v1
     kind: Job
     metadata:
       name: tensorflow-benchmark-share-mem-core
     spec:
       parallelism: 1
       template:
         metadata:
           labels:
             app: tensorflow-benchmark-share-mem-core
         spec:
           containers:
           - name: tensorflow-benchmark
             image: registry.cn-beijing.aliyuncs.com/ai-samples/gpushare-sample:benchmark-tensorflow-2.2.3
             command:
             - bash
             - run.sh
             - --num_batches=5000000
             - --batch_size=8
             resources:
               limits:
                 aliyun.com/gpu-mem: 10 # Requests 10 GiB of GPU memory.
                 aliyun.com/gpu-core.percentage: 30  # Requests 30% of the computing power of one card.
             workingDir: /root
           restartPolicy: Never

2. Run the following commands to deploy the Jobs:

   kubectl apply -f tensorflow-benchmark-exclusive.yaml
   kubectl apply -f tensorflow-benchmark-share-mem.yaml
   kubectl apply -f tensorflow-benchmark-share-mem-core.yaml

3. Run the following command to check the status of the Pods:

   kubectl get pod

   Expected output:

   NAME                                        READY   STATUS    RESTARTS   AGE
   tensorflow-benchmark-exclusive-7dff2        1/1     Running   0          3m13s
   tensorflow-benchmark-share-mem-core-k24gz   1/1     Running   0          4m22s
   tensorflow-benchmark-share-mem-shmpj        1/1     Running   0          3m46s

   The output shows that all Pods are in the Running state, which indicates that the Jobs were deployed successfully.

Step 3: View the GPU dashboard