Container Compute Service:Use GPU sharing on GPU-HPN nodes

Resource configuration and QoS

Shared GPU resources are described using Kubernetes requests/limits constraints. You can configure computing power and GPU memory as percentages. The feature also supports resource descriptions where limits are greater than requests. This may cause multiple pods to compete for GPU resources simultaneously. ACS defines a Quality of Service (QoS) for shared GPU resources. When multiple pods on a node use GPU resources simultaneously, the pods are queued and preemption may be triggered. The following is an example:

...
resources:
  requests:  # Controls the number of pods that can be scheduled on the node.
    alibabacloud.com/gpu-core.percentage: 10         # The percentage of computing power that the pod requires.
    alibabacloud.com/gpu-memory.percentage: 10       # The percentage of GPU memory that the pod requires.
  limits:    # Controls the upper limit of resources that can be used at runtime. For more information about the effects, see the configuration instructions.
    alibabacloud.com/gpu-core.percentage: 100        # The upper limit of computing power usage.
    alibabacloud.com/gpu-memory.percentage: 100      # The upper limit of GPU memory usage. Exceeding this limit causes a CUDA OOM error.
...

Similar to the process management mechanism of an operating system, the GPU sharing module classifies pods into three states: hibernation, ready, and running. The state transition process is shown in the following figure.

• When a pod starts, it enters the hibernation state.

• When the pod attempts to use GPU resources, it enters the ready state. The GPU sharing module then allocates GPU resources to the pod based on a priority policy.

• After the pod is allocated GPU resources, it enters the running state.

• If pods are still in the ready state after all resources are allocated, a preemption mechanism is triggered to ensure resource fairness among pods.

• When a pod is preempted, the process that occupies the GPU resources is killed, and the pod returns to the hibernation state.

Resource sharing models

GPU sharing is based on a shared model and allows multiple pods to run on a single GPU card simultaneously. ACS currently supports the following sharing models:

Example: Use the share-pool model for off-peak resource usage in Notebook scenarios

In Notebook development, applications typically do not occupy resources for long periods. You can use the share-pool model to allow pods to run on different GPU cards during off-peak hours. A pod enters the ready queue to wait for resources only when it requires resources.

The following is a use case for a Notebook scenario:

Pods A and B are configured with `requests=0.5` and `limits=0.5`. Pods C and D are configured with `requests=0.5` and `limits=1`. Based on the `requests` values, these pods can be scheduled to a single GPU-HPN node that has two GPUs.

Time T1:

1. Pod A and Pod C are occupying resources. Pod B and Pod D are in the ready queue, waiting to be scheduled.

2. The GPU sharing module attempts to allocate resources to Pod D, which is at the head of the queue. However, GPU 0 has only 0.5 GPU of idle resources. Although Pod D's `request` is 0.5, which the available capacity can satisfy, its `limit` is 1. Running Pod A and Pod D on the same GPU would cause resource competition. Therefore, the GPU sharing module keeps Pod D in the queue.

Time T2 - Phase 1:

1. Pod C's task is complete, and it enters the hibernation queue.

2. After GPU 1 becomes idle, its resources are allocated to Pod D.

Time T2 - Phase 2:

Pod B is allocated resources. Because Pod B's `limit` is 0.5, it can run on GPU 0 simultaneously with Pod A without resource competition.

Step 1: Add a label to the GPU-HPN node

1. View the GPU-HPN nodes.

   Important

   Before enabling this feature, you must delete any pods on the node that request exclusive GPU resources. You do not need to delete pods that request only CPU and memory resources.

   kubectl get node -l alibabacloud.com/node-type=reserved

   Expected output:

   NAME                     STATUS   ROLES   AGE   VERSION
   cn-wulanchabu-c.cr-xxx   Ready    agent   59d   v1.28.3-aliyun

2. Add the label alibabacloud.com/gpu-share-policy=share-pool to the node cn-wulanchabu-c.cr-xxx to enable the GPU sharing feature.

   $ kubectl label node cn-wulanchabu-c.cr-xxx alibabacloud.com/gpu-share-policy=share-pool

Step 2: Check the enabling status of the node

Wait for the feature to be enabled on the node, and then check the enabling status. You can see the GPU shared resources in the capacity field. In the conditions field, GPUSharePolicyValid is True. This indicates that the feature is enabled.

$ kubectl get node cn-wulanchabu-c.cr-xxx -o yaml

After the GPU sharing policy takes effect, the node status is updated. Expected output:

# The actual output may vary.
apiVersion: v1
kind: Node
spec: 
  # ...
status:
  allocatable:
    # GPU shared resource description
    alibabacloud.com/gpu-core.percentage: "1600"
    alibabacloud.com/gpu-memory.percentage: "1600"
    # After the feature is enabled, CPU, memory, and storage resources are reserved for the GPU sharing module.
    cpu: "144"
    memory: 1640Gi
    nvidia.com/gpu: "16"
    ephemeral-storage: 4608Gi
  capacity:
    # GPU shared resource description
    alibabacloud.com/gpu-core.percentage: "1600"
    alibabacloud.com/gpu-memory.percentage: "1600"
    cpu: "176"
    memory: 1800Gi
    nvidia.com/gpu: "16"
    ephemeral-storage: 6Ti
  conditions:
  # Indicates whether the GPU Share policy configuration is valid.
  - lastHeartbeatTime: "2025-01-07T04:13:04Z"
    lastTransitionTime: "2025-01-07T04:13:04Z"
    message: gpu share policy is valid.
    reason: Valied
    status: "True"
    type: GPUSharePolicyValid
  # Indicates the GPU Share policy that is in effect on the current node.
  - lastHeartbeatTime: "2025-01-07T04:13:04Z"
    lastTransitionTime: "2025-01-07T04:13:04Z"
    message: gpu share policy is share-pool.
    reason: share-pool
    status: "True"
    type: GPUSharePolicy

For more information about the configuration items for GPU shared resources, see Node configuration.

Step 3: Deploy a pod with GPU shared resource specifications

1. Create a file named gpu-share-demo.yaml. Configure it to use the same share-pool model as the node.

   apiVersion: v1
   kind: Pod
   metadata:
     labels:
       alibabacloud.com/compute-class: "gpu-hpn"
       # Set the GPU sharing model for the pod to share-pool, which is the same as the node configuration.
       alibabacloud.com/gpu-share-policy: "share-pool" # static
     name: gpu-share-demo
     namespace: default
   spec:
     containers:
     - name: demo
       image: registry-cn-wulanchabu-vpc.ack.aliyuncs.com/acs/stress:v1.0.4
       args:
         - '1000h'
       command:
         - sleep
       # Specify the GPU shared resources gpu-core.percentage and gpu-memory.percentage in the resource description.
       # For more information about the effects of request and limit, see the configuration instructions.
       resources:
         limits:
           cpu: '5'
           memory: 50Gi
           alibabacloud.com/gpu-core.percentage: 100
           alibabacloud.com/gpu-memory.percentage: 100
         requests:
           cpu: '5'
           memory: 50Gi
           alibabacloud.com/gpu-core.percentage: 10
           alibabacloud.com/gpu-memory.percentage: 10

2. Deploy the sample pod.

   kubectl apply -f gpu-share-demo.yaml

Step 4: Check the GPU shared resource usage of the pod

Log on to the container to check the GPU shared resource usage of the pod.

kubectl exec -it pod gpu-share-demo -- /bin/bash

Use commands such as `nvidia-smi` to view the GPU resource allocation and usage of the container. The actual output may vary.

(Optional) Step 5: Disable the GPU sharing policy on the node

1. Delete the pod that uses the GPU sharing feature.

   $ kubectl delete pod gpu-share-demo

2. Disable the GPU sharing feature on the node.

   $ kubectl label node cn-wulanchabu-c.cr-xxx alibabacloud.com/gpu-share-policy=none

3. Check the policy configuration status of the node again.

   $ kubectl get node cn-wulanchabu-c.cr-xxx -o yaml

   Expected output:

   apiVersion: v1
   kind: Node
   spec: 
     # ...
   status:
     allocatable:
       # After the feature is disabled, the reserved CPU and memory resources are restored to their initial values.
       cpu: "176"
       memory: 1800Gi
       nvidia.com/gpu: "16"
       ephemeral-storage: 4608Gi
     capacity:
       cpu: "176"
       memory: 1800Gi
       nvidia.com/gpu: "16"
       ephemeral-storage: 6Ti
     conditions:
     # Indicates whether the GPU Share policy configuration is valid.
     - lastHeartbeatTime: "2025-01-07T04:13:04Z"
       lastTransitionTime: "2025-01-07T04:13:04Z"
       message: gpu share policy config is valid.
       reason: Valid
       status: "True"
       type: GPUSharePolicyValid
     # Indicates the GPU Share policy that is in effect on the current node.
     - lastHeartbeatTime: "2025-01-07T04:13:04Z"
       lastTransitionTime: "2025-01-07T04:13:04Z"
       message: gpu share policy is none.
       reason: none
       status: "False"
       type: GPUSharePolicy

Node configuration

apiVersion: v1
kind: Node
metadata:
  labels:
    # Enable the feature and use the share-pool policy. You can specify other policies in the value.
    alibabacloud.com/gpu-share-policy: share-pool

apiVersion: v1
kind: Node
...
metadata:
  annotations:
    alibabacloud.com/gpu-share-qos-config: '{"preemptEnabled": true, "podMaxDurationMinutes": 120, "reservedEphemeralStorage": "1.5Ti"}'
...

Pod configuration

After the feature is enabled, you can use it by configuring the GPU shared resource label in the pod.

apiVersion: v1
kind: Pod
metadata:
  labels:
    # Only the gpu-hpn compute class is supported.
    alibabacloud.com/compute-class: "gpu-hpn"
    # Set the GPU sharing model for the pod to share-pool, which is the same as the node configuration.
    alibabacloud.com/gpu-share-policy: "share-pool"
  name: gpu-share-demo
  namespace: default
spec:
  containers:
  - name: demo
    image: registry-cn-wulanchabu-vpc.ack.aliyuncs.com/acs/stress:v1.0.4
    args:
      - '1000h'
    command:
      - sleep
    resources:
      limits:
        cpu: '5'
        memory: 50Gi
        alibabacloud.com/gpu-core.percentage: 100
        alibabacloud.com/gpu-memory.percentage: 100
      requests:
        cpu: '5'
        memory: 50Gi
        alibabacloud.com/gpu-core.percentage: 10
        alibabacloud.com/gpu-memory.percentage: 10

The configuration items are described as follows:

What happens to a pod in the ready queue if no GPU resources are available?

When a GPU sharing pod is waiting for resources, it periodically prints a message. The following is a sample message.

You have been waiting for ${1} seconds. Approximate position: ${2}

The ${1} parameter indicates the waiting time, and the ${2} parameter indicates the current position in the ready queue.

What are the pod monitoring metrics specific to the GPU sharing mode?

For pods that use GPU shared resources, you can use the following metrics to view their resource usage.

DCGM_FI_POOLING_STATUS{NodeName="cn-wulanchabu-c.cr-xxx",pod="gpu-share-demo",namespace="default"} 1

DCGM_FI_POOLING_POSITION{NodeName="cn-wulanchabu-c.cr-xxx",pod="gpu-share-demo",namespace="default"} 1

How are GPU utilization metrics different when a pod uses GPU sharing?

The GPU utilization metrics for a pod are the same as before. However, for pods that use GPU sharing, the labels and meanings of the metrics are different.

• In the pod monitoring data provided by ACS, metrics such as GPU computing power utilization and GPU memory usage are absolute values based on the entire GPU card, which is the same as in the exclusive GPU scenario.

• The GPU memory usage seen within a pod using commands such as `nvidia-smi` is an absolute value, which is the same as in the exclusive GPU scenario. However, the computing power utilization is a relative value, where the denominator is the pod's limit.

• The device information, such as the ID number in the pod's GPU utilization metrics, corresponds to the actual ID on the node. The numbering does not always start from 0.

• For the share-pool sharing model, the device number in the metrics may change because the pod elastically uses different GPU devices from the pool.

If GPU sharing is enabled on only some nodes in a cluster, how can I avoid scheduling conflicts with exclusive GPU pods?

The default scheduler in an ACS cluster automatically matches pod and node types to avoid scheduling conflicts.

If you use a custom scheduler, an exclusive GPU pod might be scheduled to a GPU sharing node because the node's capacity includes both GPU device resources and GPU shared resources. You can choose one of the following solutions:

• Solution 1: Write a scheduler plugin that automatically detects the configuration labels and condition protocol of ACS nodes to filter out nodes of a mismatched type. For more information, see Scheduling Framework.

• Solution 2: Use Kubernetes labels or taints. Add a label or taint to the nodes where GPU sharing is enabled. Then, configure different affinity policies for exclusive GPU pods and shared GPU pods.

What information is available when a GPU sharing pod is preempted?

For the share-pool sharing model, when preemption is triggered, the pod has an Event and a Condition. An Event is in an unstructured data format. To read structured data, you can retrieve it from the `reason` and `status` fields of the corresponding Condition. The details are as follows.

# Indicates that the GPU resources of the current pod were preempted. The name of the preempting pod is <new-pod-name>.
Warning  GPUSharePreempted  5m15s  gpushare   GPU is preempted by <new-pod-name>.
# Indicates that the current pod preempted the GPU resources of another pod. The name of the preempted pod is <old-pod-name>.
Warning  GPUSharePreempt    3m47s  gpushare   GPU is preempted from <old-pod-name>.

- type: Interruption.GPUShareReclaim # The event type for a GPU sharing pod preemption.
  status: "True" # True indicates that a preemption or preemption-by action occurred.
  reason: GPUSharePreempt # GPUSharePreempt indicates that this pod preempted another pod. GPUSharePreempted indicates that this pod was preempted by another pod.
  message: GPU is preempted from <old-pod-name>. # A user-friendly message similar to the event.
  lastTransitionTime: "2025-04-22T08:12:09Z" # The time when the preemption occurred.
  lastProbeTime: "2025-04-22T08:12:09Z"

How can I run more pods on a node in a Notebook scenario?

For pods with GPU sharing enabled, ACS also lets you configure CPU and memory specifications where the `request` is less than the `limit`. This helps to fully utilize node resources. Note that when the total `limit` of resources for pods submitted to a node exceeds the node's allocatable resources, the pods will compete for CPU and memory resources. You can analyze the resource competition for CPU and memory by reviewing the node's resource utilization data. For more information, see ACS GPU-HPN node-level monitoring metrics. For a pod, CPU resource competition is reflected in the pod's CPU Steal Time. Memory resource competition triggers a machine-wide out-of-memory (OOM) error, which causes some pods to be killed. Plan your pod priorities and resource specifications based on your application's characteristics to avoid affecting pod service quality due to resource competition.