How to run the ack-cgpu component in a dedicated GPU cluster -

This topic describes how to use the cGPU feature in a dedicated GPU cluster after installing the ack-cgpu component.

Prerequisites

Ensure that the ack-cgpu component is installed. For more information, see Install the ack-cgpu component.

Usage notes

For GPU nodes that are managed in Container Service for Kubernetes (ACK) clusters, you need to pay attention to the following items when you request GPU resources for applications and use GPU resources.

Do not run GPU-heavy applications directly on nodes.
Do not use tools, such as Docker, Podman, or nerdctl, to create containers and request GPU resources for the containers. For example, do not run the docker run --gpus all or docker run -e NVIDIA_VISIBLE_DEVICES=all command and run GPU-heavy applications.
Do not add the NVIDIA_VISIBLE_DEVICES=all or NVIDIA_VISIBLE_DEVICES=<GPU ID> environment variable to the env section in the pod YAML file. Do not use the NVIDIA_VISIBLE_DEVICES environment variable to request GPU resources for pods and run GPU-heavy applications.
Do not set NVIDIA_VISIBLE_DEVICES=all and run GPU-heavy applications when you build container images if the NVIDIA_VISIBLE_DEVICES environment variable is not specified in the pod YAML file.
Do not add privileged: true to the securityContext section in the pod YAML file and run GPU-heavy applications.

The following potential risks may exist when you use the preceding methods to request GPU resources for your application:

If you use one of the preceding methods to request GPU resources on a node but do not specify the details in the device resource ledger of the scheduler, the actual GPU resource allocation information may be different from that in the device resource ledger of the scheduler. In this scenario, the scheduler can still schedule certain pods that request the GPU resources to the node. As a result, your applications may compete for resources provided by the same GPU, such as requesting resources from the same GPU, and some applications may fail to start up due to insufficient GPU resources.
Using the preceding methods may also cause other unknown issues, such as the issues reported by the NVIDIA community.

Procedure

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of the target cluster. In the left-side navigation pane, choose Nodes > Node Pools.
Log on to the target master node and run the following command to query the cGPU capabilities of the cluster.
Note
- For more information about how to log on to a master node, see Connect to an instance by using VNC.
- To query the cGPU capabilities of a cluster from a local client, you must install the ack-cgpu component and a resource query tool. For more information, see Step 4: Install and use the GPU resource query tool.
```
kubectl inspect cgpu
```
Expected output:
```
NAME                     IPADDRESS    GPU0(Allocated/Total)  GPU1(Allocated/Total)  GPU Memory(GiB)
cn-beijing.192.168.XX.XX   192.168.XX.XX  0/7                    0/7                    0/14
---------------------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
0/14 (0%)
```
Note
To query detailed information about the cGPU capabilities, run the kubectl inspect cgpu -d command.

In the left-side navigation pane, choose Workloads > Jobs. On the Jobs page, click Create from YAML in the upper-right corner. On the page that appears, select a namespace and choose a template from the Sample Template drop-down list or use a custom template. Then, enter the following sample YAML into the code editor and click Create.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-share-sample
spec:
  containers:
  - name: gpu-share-sample
    image: registry.cn-hangzhou.aliyuncs.com/ai-samples/gpushare-sample:tensorflow-1.5    
    command:
    - python
    - tensorflow-sample-code/tfjob/docker/mnist/main.py
    - --max_steps=100000
    - --data_dir=tensorflow-sample-code/data
    resources:
      limits:
        # Unit: GiB. The Pod requests a total of 3 GiB of GPU memory.
        aliyun.com/gpu-mem: 3 # Specify the amount of GPU memory.
    workingDir: /root

On the master node, run the following command again to check the GPU memory usage in the cluster.

kubectl inspect cgpu

Expected output:

NAME                      IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
cn-beijing.192.168.XX.XX  192.168.XX.XX  3/14                   3/14
---------------------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
3/14 (21%)

The output indicates that the node cn-beijing.192.168.XX.XX has a total of 14 GiB of GPU memory, and 3 GiB has been allocated.

Verify the result

To verify that GPU memory isolation is enabled on the node, follow these steps.

Log on to the target master node.

Run the following command to view the application's logs and confirm that cGPU memory isolation is enabled.

kubectl logs gpu-share-sample --tail=1

Expected output:

2023-08-07 09:08:13.931003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2832 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:07.0, compute capability: 7.5)

The output indicates that the container requested 2,832 MB of GPU memory.

Run the following command to log on to the container and view the total amount of GPU memory allocated to it.

kubectl exec -it gpu-share-sample nvidia-smi

Expected output:

Mon Aug 7 08:52:18 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:07.0 Off |                    0 |
| N/A   41C    P0    26W /  70W |   3043MiB /  3231MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

The output indicates that the total amount of GPU memory allocated to this container is 3,231 MiB.

Log on to the GPU node that runs the sample application and view the node's total GPU memory.

nvidia-smi

Expected output:

Mon Aug  7 09:18:26 2023 
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:07.0 Off |                    0 |
| N/A   40C    P0    26W /  70W |   3053MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      8796      C   python3                                     3043MiB |
+-----------------------------------------------------------------------------+

The output indicates that the host node has a total of 15,079 MiB of GPU memory, of which 3,053 MiB is allocated to the container.

:Running the ack-cgpu component

Prerequisites

Usage notes

Procedure

Verify the result

Related topics