Upgrade NVIDIA drivers by using a node pool - Container Service for Kubernetes

If your CUDA library requires a newer NVIDIA driver, you must upgrade the driver version on your nodes. Using a node pool to upgrade the NVIDIA driver lets you manage different driver versions for nodes in your cluster in batches. This topic describes how to upgrade the NVIDIA driver for existing nodes by using a node pool.

Limitations

To upgrade the NVIDIA driver, you must move the node from its current node pool to a new one configured with the target driver version. You cannot upgrade a node's driver within its existing node pool, as this would affect all other nodes in that pool.

Important

During this process, the node is reimaged and the specified NVIDIA driver is installed. Before you perform the upgrade, ensure no critical workloads are running on the node and that all important data is backed up. To minimize risks, we recommend upgrading a single node first to validate the process before upgrading the rest in batches.

Considerations

ACK does not guarantee compatibility between the NVIDIA driver version and the CUDA Toolkit version. Verify compatibility before proceeding.
For driver version requirements for specific NVIDIA GPU models, see the NVIDIA official website.
For custom operating system images with pre-installed GPU components (such as the GPU driver or NVIDIA Container Runtime), ACK cannot guarantee that the pre-installed driver is compatible with other ACK GPU components, such as monitoring components.
If you customize the GPU driver version by specifying a version number or by using an OSS URL, the OS and driver may become incompatible after the update. See Supported NVIDIA driver versions in ACK to select a compatible driver.
Uploading your own GPU driver to OSS may cause incompatibilities with the OS image, ECS instance type, or container runtime, leading to node creation failure. ACK does not guarantee success with this method. Verify the configuration before proceeding.

Instance type compatibility

Certain Elastic Compute Service (ECS) instance types have driver version restrictions:

Instance type	Restriction	Recommended versions
gn7and ebmgn7	Versions 510.xxx and 515.xxx have compatibility issues	Versions earlier than 510 (e.g., 470.xxx.xxxx) with GPU System Processor (GSP) disabled, or 525.125.06 and later
ebmgn7and ebmgn7e	Minimum version required	460.32.03 or later

Use cases

If you use node pool A to manage nodes with an upgraded NVIDIA driver, you can schedule workloads to these nodes by setting the workload's nodeselector to the label of node pool A.
If you need to upgrade the NVIDIA driver to version 550.144.03 for one group of nodes and to version 535.161.07 for another, you can add the first group of nodes to node pool A and the second group to node pool B.

Step 1: Determine the NVIDIA driver version

Before you select an NVIDIA driver version, you must identify which versions are compatible with your CUDA library. Refer to the cuda-toolkit-release-notes for compatibility information and select a suitable NVIDIA driver version.

Step 2: Remove nodes

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Nodes.
Select the nodes whose drivers you want to upgrade and click Batch Remove. In the Remove Node dialog box, select Drain Node and then click OK.

Step 3: Create node pool and specify driver version

Use a supported driver

Note

Select an NVIDIA driver version that meets your business requirements from the list of supported versions. This topic uses driver version 550.144.03 as an example.
This installation method is simple. Add the ack.aliyun.com/nvidia-driver-version=<driver_version> label to the node pool, and then add the nodes that you removed from the cluster in Step 2 to this node pool.

Log on to the ACK console . In the left navigation pane, click Clusters.
On the Clusters page, click the cluster name. In the left navigation pane, choose Nodes > Node Pools.
In the upper-left corner, click Create Node Pool. For details about configuration parameters, see Create and manage node pools.
Under Advanced Options, go to the Node Labels section and click the icon to add a label. Set the Key to ack.aliyun.com/nvidia-driver-version and the Value to 550.144.03.

Use a custom driver

This section uses driver version 515.86.01 as an example.

Part 1: Prepare a custom driver

If the list of supported versions does not contain the driver version that you need, download the target driver version from the official NVIDIA website.
Download NVIDIA Fabric Manager from the official NVIDIA YUM repository. The NVIDIA Fabric Manager version must match the driver version.
```
wget https://developer.download.nvidia.cn/compute/cuda/repos/rhel7/x86_64/nvidia-fabric-manager-515.86.01-1.x86_64.rpm
```
Log on to the OSS console and create a bucket. For more information, see Create buckets.
We recommend that you create the OSS bucket in the same region as your ACK cluster. This allows ACK nodes to pull the driver over the internal network during installation.
Upload the NVIDIA-Linux-x86_64-515.86.01.run and nvidia-fabric-manager-515.86.01-1.x86_64.rpm files to the root directory of the bucket.
On the bucket page, go to File Management > Objects in the navigation pane on the left. In the row of the uploaded file, click Details in the Actions column.
In the Details panel, turn off the Use HTTPS switch.
On the bucket details page, click Overview in the navigation pane on the left and obtain the internal endpoint from the lower section of the page.

Part 2: Create the node pool

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.

In the upper-left corner, click Create Node Pool. For more information about the configuration parameters, see Create and manage a node pool. The following parameters are relevant for this procedure.

In the Node Labels section, click the icon to add labels.

Key	Value
`ack.aliyun.com/nvidia-driver-oss-endpoint`	The OSS Bucket internal endpoint that you obtained in Step 7 of Step 1. `my-nvidia-driver.oss-cn-beijing-internal.aliyuncs.com`
`ack.aliyun.com/nvidia-driver-runfile`	The name of the NVIDIA driver that you downloaded in Step 1. `NVIDIA-Linux-x86_64-515.86.01.run`
`ack.aliyun.com/nvidia-fabricmanager-rpm`	The name of the NVIDIA Fabric Manager that you downloaded in Substep 2 of Step 1. `nvidia-fabric-manager-515.86.01-1.x86_64.rpm`

Step 4: Add nodes to the node pool

After the node pool is created, add the nodes that you removed in Step 2 to the new node pool.

Step 5: Verify the driver upgrade

Run the following command to list the pods that have the component: nvidia-device-plugin label:

kubectl get po -n kube-system -l component=nvidia-device-plugin -o wide

Expected output:

NAME                                            READY   STATUS    RESTARTS   AGE   IP              NODE                       NOMINATED NODE   READINESS GATES
nvidia-device-plugin-cn-beijing.192.168.1.127   1/1     Running   0          6d    192.168.1.127   cn-beijing.192.168.1.127   <none>           <none>
nvidia-device-plugin-cn-beijing.192.168.1.128   1/1     Running   0          17m   192.168.1.128   cn-beijing.192.168.1.128   <none>           <none>
nvidia-device-plugin-cn-beijing.192.168.8.12    1/1     Running   0          9d    192.168.8.12    cn-beijing.192.168.8.12    <none>           <none>
nvidia-device-plugin-cn-beijing.192.168.8.13    1/1     Running   0          9d    192.168.8.13    cn-beijing.192.168.8.13    <none>           <none>
nvidia-device-plugin-cn-beijing.192.168.8.14    1/1     Running   0          9d    192.168.8.14    cn-beijing.192.168.8.14    <none>           <none>

The pod name for the newly added node, nvidia-device-plugin-cn-beijing.192.168.1.128, appears in the NODE column.

Run the following command to check whether the driver version on the node matches the target version:

kubectl exec -ti nvidia-device-plugin-cn-beijing.192.168.1.128 -n kube-system -- nvidia-smi