Customize GPU drivers on nodes by using an OSS URL - Container Service for Kubernetes

ACK clusters include a default NVIDIA GPU driver. If your application requires a specific or newer version, you can install a custom driver. This topic explains how to use node pool labels and an Object Storage Service (OSS) URL to install a custom NVIDIA driver on your GPU nodes.

Usage notes

ACK does not guarantee compatibility between GPU driver versions and CUDA library versions. You are responsible for verifying their compatibility.
For detailed driver requirements for different NVIDIA GPU models, see the official NVIDIA documentation.
If you use a custom OS image with a pre-installed GPU driver, NVIDIA Container Runtime, or other GPU components, ACK cannot guarantee that the custom GPU driver is compatible with other ACK GPU components, such as monitoring agents.
When you specify a GPU driver version by using a node pool label, the driver is installed only on new nodes added to the pool. Existing nodes are not affected. To apply the new driver to existing nodes, you must remove the nodes and then add the existing nodes back to the cluster.
The gn7 and ebmgn7 instance types have compatibility issues with driver versions 510.xxx and 515.xxx. Use a driver version earlier than 510 with GPU System Processor (GSP) disabled (for example, 470.xxx.xxxx) or version 525.125.06 or later.
ECS instances of the ebmgn7 or ebmgn7e instance type support only NVIDIA driver versions 525.125.06 or later.
If you customize the GPU driver version for your node pool by specifying a version number or by using an OSS URL, the OS and the driver may become incompatible after the OS image is updated. See Supported NVIDIA driver versions in ACK to select a compatible driver.
If you upload your own GPU driver to OSS, you may encounter incompatibilities with the OS image, ECS instance type, or container runtime, which can cause node creation to fail. ACK does not guarantee that nodes can be added successfully with this method. You are responsible for verifying the configuration.

Step 1: Download the target driver

If the list of NVIDIA driver versions supported by ACK does not include your required version, download the driver from the official NVIDIA website. This topic uses version 550.90.07 as an example. Download the NVIDIA-Linux-x86_64-550.90.07.run driver file to your local machine.

Step 2: Download NVIDIA Fabric Manager

Download NVIDIA Fabric Manager from the official NVIDIA YUM repository. The NVIDIA Fabric Manager version must match the driver version.

wget https://developer.download.nvidia.cn/compute/cuda/repos/rhel7/x86_64/nvidia-fabric-manager-550.90.07-1.x86_64.rpm

Step 3: Create an OSS bucket

Log on to the Object Storage Service (OSS) console and create an OSS bucket. For more information, see Create buckets.

Note

Create the bucket in the same region as your ACK cluster. This allows ACK nodes to pull the driver from the bucket over the internal network when installing the GPU driver.

Step 4: Upload files to the OSS bucket

Log on to the OSS console and upload the NVIDIA-Linux-x86_64-550.90.07.run and nvidia-fabric-manager-550.90.07-1.x86_64.rpm files to the root directory of the bucket.
Important
Make sure to upload the files to the root directory of the bucket, not a subdirectory.
On the bucket page, in the navigation pane on the left, click File Management > Objects. In the Actions column for the uploaded file, click Details.
In the Details panel, turn off the Use HTTPS switch.
Important
ACK pulls the driver file using an HTTP URL. However, OSS uses HTTPS by default. Therefore, you must turn off the Use HTTPS switch.
On the bucket details page, click Overview in the navigation pane on the left. Find and copy the internal endpoint from the lower part of the page.
Important
- Pulling driver files from an external endpoint is slow and can cause GPU node creation to fail. Use an internal endpoint (which contains -internal) or an accelerated domain name (which contains oss-accelerate).
- If a file download fails, see Access control to adjust the bucket's access control policy.

Step 5: Configure node pool labels

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.

Click Create Node Pool in the upper-left corner and add GPU nodes. For more information about the parameters, see Create and manage a node pool. The following parameters are key to this configuration.

In the Node Labels section, click the icon to add the following labels. Replace the example values with your actual values.

Key	Value
`ack.aliyun.com/nvidia-driver-oss-endpoint`	The internal endpoint of the OSS bucket from Step 4. `my-nvidia-driver.oss-cn-beijing-internal.aliyuncs.com`
`ack.aliyun.com/nvidia-driver-runfile`	The name of the NVIDIA driver file from Step 1. `NVIDIA-Linux-x86_64-550.90.07.run`
`ack.aliyun.com/nvidia-fabricmanager-rpm`	The name of the NVIDIA Fabric Manager file from Step 2. `nvidia-fabric-manager-550.90.07-1.x86_64.rpm`

Step 6: Verify the driver installation

Run the following command to view Pods with the component: nvidia-device-plugin label.

kubectl get po -n kube-system -l component=nvidia-device-plugin -o wide

Expected output:

NAME                                            READY   STATUS    RESTARTS   AGE   IP              NODE                       NOMINATED NODE   READINESS GATES
nvidia-device-plugin-cn-beijing.192.168.1.127   1/1     Running   0          6d    192.168.1.127   cn-beijing.192.168.1.127   <none>           <none>
nvidia-device-plugin-cn-beijing.192.168.1.128   1/1     Running   0          17m   192.168.1.128   cn-beijing.192.168.1.128   <none>           <none>
nvidia-device-plugin-cn-beijing.192.168.8.12    1/1     Running   0          9d    192.168.8.12    cn-beijing.192.168.8.12    <none>           <none>
nvidia-device-plugin-cn-beijing.192.168.8.13    1/1     Running   0          9d    192.168.8.13    cn-beijing.192.168.8.13    <none>           <none>

The expected output indicates that the Pod name for the newly added node in the NODE column is nvidia-device-plugin-cn-beijing.192.168.1.128.

Run the following command to verify that the correct driver version is installed on the node.

kubectl exec -ti nvidia-device-plugin-cn-beijing.192.168.1.128 -n kube-system -- nvidia-smi

Expected output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P100-PCIE-16GB           On  |   00000000:00:08.0 Off |                  Off |
| N/A   31C    P0             26W /  250W |       0MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

The output indicates that the driver version is 550.90.07. This confirms that the custom NVIDIA driver was successfully installed.

Other methods

When you use the CreateClusterNodePool API operation to create a node pool, you can set the OSS URL for the custom driver in the node pool configuration. The following example shows the required configuration:

{
  // Other sections are omitted.
  ......
    "tags": [
      {
        "key": "ack.aliyun.com/nvidia-driver-oss-endpoint",
        "value": "my-nvidia-driver.oss-cn-beijing-internal.aliyuncs.com"
      },
      {
        "key": "ack.aliyun.com/nvidia-driver-runfile",
        "value": "NVIDIA-Linux-x86_64-550.90.07.run"
      },
      {
        "key": "ack.aliyun.com/nvidia-fabricmanager-rpm",
        "value": "nvidia-fabric-manager-550.90.07-1.x86_64.rpm"
      }
    ],
  // Other sections are omitted.
  ......
}