Upgrade the cGPU version on a node - Container Service for Kubernetes - Alibaba Cloud - Container Service for Kubernetes

GPU sharing and scheduling in a Container Service for Kubernetes (ACK) cluster requires cGPU on every GPU node. Upgrade the cGPU version on a node to keep GPU sharing and scheduling compatible with the NVIDIA driver, kernel, and instance family of the node.

Prerequisites

cGPU is installed on the node that you want to upgrade.
kubectl is installed and configured to access the cluster.
The cGPU version that you want to install is compatible with the node. For details, see cGPU version compatibility.

Step 1: Upgrade the cGPU add-on

The add-on to upgrade depends on the cluster type. Upgrade the add-on that applies to your cluster.

Cluster type	Add-on upgrade method
ACK managed cluster Pro ACK Edge cluster Pro	Upgrade the ack-ai-installer add-on. For instructions, see Upgrade the GPU sharing component.
ACK dedicated cluster	Upgrade the ack-cgpu add-on in the ACK console, as described in the following procedure.

To upgrade the ack-cgpu add-on in an ACK dedicated cluster

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Helm.
On the Helm page, find the ack-cgpu add-on and click Update. Select the required Version and click OK.

Step 2: Upgrade cGPU on existing nodes

After you upgrade the add-on, upgrade the existing GPU nodes so that they run the new cGPU version. Note the following points before you upgrade a node:

GPU workloads — Stop all GPU workloads on the node for the duration of the upgrade.
Rollout order — Upgrade one node first. After you verify that the GPU workloads run as expected, upgrade the remaining GPU nodes in batches.

Method 1: Remove and re-add the node

Warning

Method 1 resets the system disk of the node. If the system disk of the node contains data, back up the data before you remove the node.

Remove the node and add it back

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Nodes.
On the Nodes page, select the cGPU node that you want to upgrade and click Batch Remove. In the Remove Node dialog box, select Drain Node.
Re-add the removed node to its original node pool. For instructions, see Add existing nodes.
Important
Add the node in automatic mode. The manual mode does not reset the node.

Verify the cGPU version

Run the following command to check the status of the cgpu-installer Pod on the node that you added back.

kubectl get po -l name=cgpu-installer -n kube-system -o wide

Example output:

NAME                   READY   STATUS    RESTARTS   AGE    IP                NODE                         NOMINATED NODE   READINESS GATES
cgpu-installer-*****   1/1     Running   0          4d2h   192.168.XXX.XX1   cn-beijing.192.168.XXX.XX1   <none>           <none>
cgpu-installer-**2     1/1     Running   0          4d2h   192.168.XXX.XX2   cn-beijing.192.168.XXX.XX2   <none>           <none>
cgpu-installer-**3     1/1     Running   0          4d2h   192.168.XXX.XX3   cn-beijing.192.168.XXX.XX3   <none>           <none>

Run the following command to access the cgpu-installer Pod. Replace cgpu-installer-xxxxx with the Pod name in the preceding output.
```
kubectl exec -ti cgpu-installer-xxxxx -n kube-system -- bash
```
Run the following command to check the current cGPU version.
```
nsenter -t 1 -i -p -n -u -m -- cat /proc/cgpu_km/version
```
Example output:
```
1.5.16
```
Note
For information about the latest cGPU version, see ack-ai-installer.

cGPU version compatibility

A cGPU version must be compatible with the NVIDIA driver, instance family, nvidia-container-toolkit, and kernel version of the node. Check the cGPU version that you want to install against each of the following tables.

NVIDIA driver compatibility

cGPU version	Compatible NVIDIA drivers
1.5.20, 1.5.19, 1.5.18, 1.5.17, 1.5.16, 1.5.15, 1.5.13, 1.5.12, 1.5.11, 1.5.10, 1.5.9, 1.5.8, 1.5.7	Supported: 460 series; 470 series; 510 series; 515 series; 525 series; 535 series; 550 series; 560 series; 565 series; 570 series; 575 series.
1.5.6, 1.5.5, 1.5.3, 1.5.2, 1.0.10, 1.0.9, 1.0.8, 1.0.7, 1.0.6, 1.0.5	Supported: 460 series; 470 series <= 470.161.03; 510 series <= 510.108.03; 515 series <= 515.86.01; 525 series <= 525.89.03. Not supported: 535 series; 550 series; 560 series; 565 series; 570 series; 575 series.
1.0.3, 0.8.17, 0.8.13	Supported: 460 series; 470 series <= 470.161.03. Not supported: 510 series; 515 series; 525 series; 535 series; 550 series; 560 series; 565 series; 570 series; 575 series.

Instance family compatibility

The following table lists compatibility by instance family rather than by individual instance type.

cGPU version	Compatible instance families
1.5.20, 1.5.19	Supported: gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e; gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e; gn8t / ebmgn8t; gn8is / gn8v / ebmgn8is / ebmgn8v; gn8ia / ebmgn8ia; ebmgn9t.
1.5.18, 1.5.17, 1.5.16, 1.5.15, 1.5.13, 1.5.12, 1.5.11, 1.5.10, 1.5.9	Supported: gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e; gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e; gn8t / ebmgn8t; gn8is / gn8v / ebmgn8is / ebmgn8v; gn8ia / ebmgn8ia. Not supported: ebmgn9t.
1.5.8, 1.5.7	Supported: gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e; gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e; gn8t / ebmgn8t; gn8is / gn8v / ebmgn8is / ebmgn8v. Not supported: gn8ia / ebmgn8ia; ebmgn9t.
1.5.6, 1.5.5, 1.5.3, 1.5.2, 1.0.10, 1.0.9, 1.0.8, 1.0.7, 1.0.6, 1.0.5, 1.0.3	Supported: gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e; gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e. Not supported: gn8t / ebmgn8t; gn8is / gn8v / ebmgn8is / ebmgn8v; gn8ia / ebmgn8ia; ebmgn9t.
0.8.17, 0.8.13	Supported: gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e. Not supported: gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e; gn8t / ebmgn8t; gn8is / gn8v / ebmgn8is / ebmgn8v; gn8ia / ebmgn8ia; ebmgn9t.

nvidia-container-toolkit compatibility

cGPU version	Compatible nvidia-container-toolkit versions
1.5.20, 1.5.19, 1.5.18, 1.5.17, 1.5.16, 1.5.15, 1.5.13, 1.5.12, 1.5.11, 1.5.10, 1.5.9, 1.5.8, 1.5.7, 1.5.6, 1.5.5, 1.5.3, 1.5.2, 1.0.10	Supported: <= 1.10; 1.11 to 1.17.
1.0.9, 1.0.8, 1.0.7, 1.0.6, 1.0.5, 1.0.3, 0.8.17, 0.8.13	Supported: <= 1.10. Not supported: 1.11 to 1.17.

Kernel version compatibility

cGPU version	Compatible kernel versions
1.5.20, 1.5.19, 1.5.18, 1.5.17, 1.5.16, 1.5.15, 1.5.13, 1.5.12, 1.5.11, 1.5.10, 1.5.9	Supported: kernel 3.x; kernel 4.x; kernel 5.x <= 5.15.
1.5.8	Supported: kernel 3.x; kernel 4.x; kernel 5.x <= 5.10.
1.5.7, 1.5.6, 1.5.5, 1.5.3, 1.5.2, 1.0.10, 1.0.9, 1.0.8, 1.0.7, 1.0.6, 1.0.5, 1.0.3	Supported: kernel 3.x; kernel 4.x; kernel 5.x <= 5.1.
0.8.17	Supported: kernel 3.x; kernel 4.x; kernel 5.x <= 5.0.
0.8.13, 0.8.12, 0.8.10	Supported: kernel 3.x; kernel 4.x. Not supported: kernel 5.x.