Manage cGPU by using node pools - - Alibaba Cloud Documentation Center

Scenarios

The features described in this topic are supported only in dedicated clusters, not in managed clusters.
To install the ack-cgpu component in an ACK Pro cluster, see Manage the ack-cgpu component.

Prerequisites

Ensure you have completed the following tasks:

Install the ack-cgpu component.

Plan your node pools.

You can use custom names for your node pools. This topic uses cgpu and cgpu-no-isolation as examples.

Node pool	GPU sharing	Memory isolation	Labels
cgpu	Enabled	Enabled	cgpu=true cgpu.disable.isolation=false
cgpu-no-isolation	Enabled	Disabled	cgpu=true cgpu.disable.isolation=true

Background

When using GPU sharing in Container Service for Kubernetes (ACK), you might encounter the following scenarios:

For Training Job A, the application code already specifies the usable GPU memory. Therefore, the cluster only needs to provide GPU sharing, not memory isolation.
For Training Job B, the application code does not specify the usable GPU memory. In this case, the cluster must provide both GPU sharing and memory isolation.

How can you support both scenarios in a single cluster?

Using node pools to manage cGPU supports both scenarios. You just need to create two node pools:

A node pool that provides only GPU sharing, not memory isolation. This pool is for jobs like Training Job A.
A node pool that provides both GPU sharing and memory isolation. This pool is for jobs like Training Job B.

Usage notes

When using node pools to manage cGPU, keep the following in mind:

If a job does not have a nodeSelector specified, its pods can be scheduled to any node pool, which may lead to unexpected results.

Important
Always specify a nodeSelector for each job.

If a node label changes (for example, from cgpu.disable.isolation=false to cgpu.disable.isolation=true), you must restart the gpushare-device-plugin pod on that node for the memory isolation configuration to take effect.

To restart the plugin, delete the existing gpushare-device-plugin pod. ACK then automatically creates a new one. Follow these steps:

Run the following command to list the gpushare-device-plugin pods in the cluster:

kubectl get po -n kube-system -l name=gpushare-device-plugin-ds -o wide

The following output is expected:

NAME                              READY   STATUS    RESTARTS   AGE   IP              NODE                        NOMINATED NODE   READINESS GATES
gpushare-device-plugin-ds-6r8gs   1/1     Running   0          18h   192.168.7.157   cn-shanghai.192.168.7.157   <none>           <none>
gpushare-device-plugin-ds-pjrvn   1/1     Running   0          15h   192.168.7.158   cn-shanghai.192.168.7.158   <none>           <none>

For example, to delete the pod on the cn-shanghai.192.168.7.157 node, run the following command:
```
kubectl delete po gpushare-device-plugin-ds-6r8gs -n kube-system
```

Step 1: Create node pools

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.
In the upper-right corner, click Create Node Pool.
On the Create Node Pool page, configure the parameters for the node pool.
For more information about the parameters, see Create an ACK managed cluster. The following list describes some key parameters:
- Quantity: The initial number of nodes in the node pool. Set this to 0 if you are not creating nodes now.
- Operating System: Select the operating system for the nodes, such as CentOS 7.x or Alibaba Cloud Linux 2.x.
- Node Labels: Labels to apply to the nodes in this node pool.
- ECS Tags: Tags to apply to the underlying ECS instances.
- Custom Resource Group: The resource group for the nodes in the node pool.
In the Node Labels section, add specific labels to each node pool.
- cgpu node pool: cgpu=true and cgpu.disable.isolation=false
- cgpu-no-isolation node pool: cgpu=true and cgpu.disable.isolation=true
The following configuration uses the cgpu-no-isolation node pool as an example.
Click Confirm.

On the Node Pools page, a Status of Initializing indicates the node pool is being created. When creation is complete, the Status changes to Active.

Note

If you need to add GPU nodes, you can scale out the node pool. For more information, see Create and manage node pools.

Step 2: Submit jobs

Submit two jobs, cgpu-test and cgpu-test-no-isolation. You must specify a nodeSelector in the YAML file for each job.

cgpu-test: The available GPU memory size is not set in this job's code, so it requires cGPU memory isolation to run correctly. The following sample YAML file shows the configuration:

apiVersion: batch/v1
kind: Job
metadata:
  name: cgpu-test
spec:
  parallelism: 1
  template:
    metadata:
      labels:
        app: cgpu-test
    spec:
      nodeSelector:
        cgpu.disable.isolation: "false" # Add a nodeSelector to select the cgpu node pool.
      containers:
      - name: cgpu-test
        image: registry.cn-hangzhou.aliyuncs.com/ai-samples/gpushare-sample:tensorflow-1.5
        command:
        - python
        - tensorflow-sample-code/tfjob/docker/mnist/main.py
        - --max_steps=100000
        - --data_dir=tensorflow-sample-code/data
        resources:
          limits:
            # This pod requests a total of 3 GiB of GPU memory.
            aliyun.com/gpu-mem: 3
        workingDir: /root
      restartPolicy: Never

Note

nodeSelector: Specifies the cgpu node pool.
cgpu.disable.isolation: "false": Schedules the job to a node in the cgpu node pool.
aliyun.com/gpu-mem: Sets the amount of GPU memory in GiB.

cgpu-test-no-isolation: This job's code manages its own GPU memory usage, so it does not require cGPU memory isolation. The following sample YAML file shows the configuration:

apiVersion: batch/v1
kind: Job
metadata:
  name: cgpu-test-no-isolation
spec:
  parallelism: 1
  template:
    metadata:
      labels:
        app: cgpu-test-no-isolation
    spec:
      nodeSelector:
        cgpu.disable.isolation: "true" # Add a nodeSelector to select the cgpu-no-isolation node pool.
      containers:
      - name: cgpu-test-no-isolation
        image: registry.cn-hangzhou.aliyuncs.com/ai-samples/gpushare-sample:tensorflow-1.5
        command:
        - python
        - tensorflow-sample-code/tfjob/docker/mnist/main.py
        - --max_steps=100000
        - --data_dir=tensorflow-sample-code/data
        resources:
          limits:
            # This pod requests a total of 3 GiB of GPU memory.
            aliyun.com/gpu-mem: 3

Note

nodeSelector: Specifies the cgpu-no-isolation node pool.
cgpu.disable.isolation: "true": Schedules the job to a node in the cgpu-no-isolation node pool.
aliyun.com/gpu-mem: Sets the amount of GPU memory in GiB.

Step 3: Verify the results

Run the following command to check the job status:

kubectl get po

The following output is expected:

NAME                       READY   STATUS    RESTARTS   AGE
cgpu-test-0                1/1     Running   0          5m55s
cgpu-test-no-isolation-0   1/1     Running   0          6m42s

Run the nvidia-smi command in the cgpu-test-0 pod, which requires memory isolation, to check the GPU memory available to the container.

kubectl exec cgpu-test-0 -- nvidia-smi

The following output is expected:

Mon Nov  2 11:33:10 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:07.0 Off |                    0 |
| N/A   34C    P0    54W / 300W |   3039MiB /  3226MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

The total memory visible to the container is 3,226 MiB, far less than the physical card's 16 GiB total. This confirms that cGPU memory isolation is in effect.

Run the nvidia-smi command in the cgpu-test-no-isolation-0 pod, which does not require memory isolation, to check the GPU memory available to the container.

kubectl exec cgpu-test-no-isolation-0 -- nvidia-smi

The following output is expected:

Mon Nov  2 11:39:59 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:07.0 Off |                    0 |
| N/A   37C    P0    56W / 300W |   1929MiB / 16130MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

The total memory visible to the container is 16,130 MiB, which is the total memory of the 16 GiB GPU card. This confirms that cGPU memory isolation is disabled. In this scenario, the application in the container must determine its allowed memory quota from the following environment variables. Run the following command to query the allowed GPU memory size.

kubectl exec cgpu-test-no-isolation-0 -- env | grep ALIYUN

The following output is expected:

ALIYUN_COM_GPU_MEM_CONTAINER=3    # The amount of memory in GiB that this container is allowed to use on a GPU card. In this case, 3 GiB.
ALIYUN_COM_GPU_MEM_DEV=15      # The total memory of the GPU card.
...

Compare the nvidia-smi output from the cgpu-test-no-isolation-0 and cgpu-test-0 pods.

The output for cgpu-test-no-isolation-0 shows the entire GPU card's memory, while the output for cgpu-test-0 shows only its allocated slice of memory. This demonstrates that node pools provide an effective way to manage cGPU capabilities.