How to use GPU sharing in an ACK Lingjun managed cluster - Container Service for Kubernetes

GPU sharing lets multiple pods run on the same GPU card in an ACK Lingjun managed cluster. Depending on whether your workloads need strict memory boundaries, choose between two modes:

Mode	Node label	How it works	GPU memory isolated?	Use when
Sharing without isolation	`ack.node.gpu.schedule=share`	Pods share the GPU; memory is not partitioned between them	No	Workloads that manage their own memory limits (for example, Java apps with `-Xmx`)
Sharing with isolation (eGPU)	`ack.node.gpu.schedule=egpu_mem` for GPU memory isolation only, or `ack.node.gpu.schedule=egpu_core_mem` for GPU memory isolation and computing power isolation	Pods share the GPU; each pod gets a hard memory boundary enforced by the eGPU module	Yes	Multiple containers on one GPU where one must not starve others

GPU sharing in a Lingjun node pool takes effect only on nodes that carry one of these labels.

Prerequisites

Before you begin, ensure that you have:

An ACK Lingjun managed cluster with at least one GPU-accelerated Lingjun node. See Create a Lingjun cluster with ACK activated
A Lingjun node pool. See Create a Lingjun node pool.
The KuberGPU (eGPU) capability enabled in your Lingjun bare metal cluster.
The ack-ai-installer add-on installed in your ACK managed Pro cluster. When you Deploy the cloud-native AI suite, select Scheduling Policy Extension (Batch Task Scheduling, GPU Sharing, Topology-aware GPU Scheduling) to install the component.
The GPU sharing add-on installed by default in ACK Lingjun managed clusters. To enable GPU sharing on a node, add a GPU sharing label to the node.

Enable GPU sharing without isolation

Use this mode when your workloads provide their own GPU memory limits. In this case, using the GPU isolation module to isolate application memory causes problems such as resource conflicts, so shared GPU scheduling supports not installing the isolation module on these nodes. In GPU sharing without isolation, multiple pods run on the same GPU card. Interference between the pods on the card, such as contention for GPU memory, is not handled or is left to the application layer.

Step 1: Label the node

Confirm the node is a Lingjun node by checking whether /etc/lingjun_metadata exists on the node. If the file exists, run nvidia-smi to verify that the GPU is available. If the output is normal, the node is a Lingjun node that supports GPU sharing and you can proceed to the next step. If the file does not exist, the node is not a Lingjun node, so GPU sharing cannot be enabled for it. Create a Lingjun node first. See Overview of Lingjun node pools.
Add the GPU sharing label to the node:
```
kubectl label node <NODE_NAME> ack.node.gpu.schedule=share
```
Alternatively, set the label in the console by using the node labels feature.

Step 2: Submit a GPU-sharing job

Create a file named tensorflow.yaml with the following content:

apiVersion: batch/v1
kind: Job
metadata:
  name: tensorflow-mnist-share
spec:
  parallelism: 1
  template:
    metadata:
      labels:
        app: tensorflow-mnist-share
    spec:
      containers:
      - name: tensorflow-mnist-share
        image: registry.cn-beijing.aliyuncs.com/ai-samples/gpushare-sample:tensorflow-1.5
        command:
        - python
        - tensorflow-sample-code/tfjob/docker/mnist/main.py
        - --max_steps=100000
        - --data_dir=tensorflow-sample-code/data
        resources:
          limits:
            aliyun.com/gpu-mem: 4  # Request 4 GiB of GPU memory
        workingDir: /root
      restartPolicy: Never

The key field is aliyun.com/gpu-mem: 4 under resources.limits, which requests 4 GiB of GPU memory for the pod.

Submit the job:
```
kubectl apply -f tensorflow.yaml
```

Step 3: Verify GPU sharing without isolation

Get the pod name:
```
kubectl get pod | grep tensorflow
```

Run nvidia-smi inside the pod:

kubectl exec -ti tensorflow-mnist-share-xxxxx -- nvidia-smi

Expected output:

Wed Jun 14 06:45:56 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.105.01   Driver Version: 515.105.01   CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:09.0 Off |                    0 |
| N/A   35C    P0    59W / 300W |    334MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

Key fields to check:

Memory-Usage shows 334MiB / 16384MiB — the pod sees the full 16,384 MiB of GPU memory, not just the 4 GiB it requested. This confirms isolation is not active. The example uses a V100 GPU card and the pod requests 4 GiB of GPU memory. The actual values depend on your environment.
If the GPU isolation module were installed, the memory field would show only the requested 4 GiB.

This mode does not enforce memory limits at the GPU driver level. Your application reads the GPU memory it can use from two environment variables, which the following output shows. To express the allocation as a percentage of the card's total memory, divide the two values: 4 / 16 = 0.25 (25% of total GPU memory).

ALIYUN_COM_GPU_MEM_CONTAINER=4   # GPU memory allocated to this pod (GiB)
ALIYUN_COM_GPU_MEM_DEV=16        # Total GPU memory per card (GiB)

Enable GPU sharing with isolation (eGPU)

Use this mode to enforce hard GPU memory boundaries between pods on the same GPU card. Industry solutions such as NVIDIA vGPU, MPS, vCUDA, and eGPU make it possible to use a GPU at finer granularity, and this section uses the eGPU isolation module as an example. In GPU sharing with isolation, multiple pods run on the same GPU card, and interference between the pods on that card is also handled.

Step 1: Label the node

Confirm the node is a Lingjun node by checking whether /etc/lingjun_metadata exists on the node. If the file exists, run nvidia-smi to verify the GPU is accessible. If the file does not exist, the node is not a Lingjun node and you cannot enable GPU sharing for it. Create a Lingjun node pool first. See Overview of Lingjun node pools.
Add the GPU sharing label to the node. Choose the label value based on which isolation you need:
Label value
What it enables
egpu_mem
GPU memory isolation only
egpu_core_mem
GPU memory isolation and computing power isolation
To enable memory isolation:
```
kubectl label node <NODE_NAME> ack.node.gpu.schedule=egpu_mem
```
You can also set the label in the console by using the node labels feature.
You can request GPU memory alone, but if you request GPU computing power you must request both GPU memory and GPU computing power. Requesting computing power alone is not supported.

Step 2: Confirm node resources are ready

After labeling the node, wait for the node to report its GPU resources, then verify:

kubectl get node <NODE_NAME> -oyaml

Look for aliyun.com/gpu-mem and aliyun.com/gpu-count in the allocatable and capacity sections:

allocatable:
  aliyun.com/gpu-count: "1"
  aliyun.com/gpu-mem: "80"
  ...
  nvidia.com/gpu: "0"
  ...
capacity:
  aliyun.com/gpu-count: "1"
  aliyun.com/gpu-mem: "80"
  ...
  nvidia.com/gpu: "0"
  ...

Key fields to check:

aliyun.com/gpu-count: "1" — the node has one GPU card.
aliyun.com/gpu-mem: "80" — the node has 80 GB of total GPU memory.
nvidia.com/gpu: "0" — the whole GPU is not exposed as an independent schedulable resource; memory is allocated via aliyun.com/gpu-mem.

To schedule a pod to an entire GPU device, add the label ack.gpushare.placement=require-whole-device to the pod. By default, the pod is then scheduled to a whole GPU card that has that amount of GPU memory. Specify the memory amount using aliyun.com/gpu-mem.

Step 3: Run a benchmarking job to verify isolation

Create a file named benchmark.yaml with the following content:

apiVersion: batch/v1
kind: Job
metadata:
  name: benchmark-job
spec:
  parallelism: 1
  template:
    spec:
      containers:
      - name: benchmark-job
        image: registry.cn-beijing.aliyuncs.com/ai-samples/gpushare-sample:benchmark-tensorflow-2.2.3
        command:
        - bash
        - run.sh
        - --num_batches=500000000
        - --batch_size=8
        resources:
          limits:
            aliyun.com/gpu-mem: 10  # Request 10 GB of GPU memory
        workingDir: /root
      restartPolicy: Never
      hostNetwork: true
      tolerations:
        - operator: Exists

Submit the job:
```
kubectl apply -f benchmark.yaml
```
After the pod starts, open a shell in the pod:
```
kubectl exec -ti benchmark-job-xxxx bash
```

Run vgpu-smi to check the GPU isolation status:

Memory-Usage shows 8307MiB / 10782MiB — the pod is capped at approximately 10 GB, confirming that GPU memory isolation is active.
Unlike nvidia-smi in the non-isolated mode, vgpu-smi shows only the memory allocated to this pod, not the total GPU memory.

vgpu-smi

Expected output:

+------------------------------------------------------------------------------+
|    VGPU_SMI 460.91.03     DRIVER_VERSION: 460.91.03     CUDA Version: 11.2   |
+-------------------------------------------+----------------------------------+
| GPU  Name                Bus-Id           |        Memory-Usage     GPU-Util |
|===========================================+==================================|
|   0  xxxxxxxx            00000000:00:07.0 |  8307MiB / 10782MiB   100% /  100% |
+-------------------------------------------+----------------------------------+

Key fields to check:

FAQ

How do I check whether the GPU sharing component is installed?

Run the following command:

kubectl get ds -nkube-system | grep gpushare

If the component is installed, the output lists the following DaemonSets:

NAME                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                    AGE
gpushare-egpu-device-plugin-ds       0         0         0       0            0           <none>
gpushare-egpucore-device-plugin-ds   0         0         0       0            0           <none>

What's next

Labels for enabling GPU scheduling policies — learn about all available node labels for GPU scheduling.
Overview of Lingjun node pools — add Lingjun nodes to your cluster.

Label value	What it enables
`egpu_mem`	GPU memory isolation only
`egpu_core_mem`	GPU memory isolation and computing power isolation