How to upgrade the cGPU module on a node by using commands and the console - Container Service for Kubernetes

Nodes in an ACK cluster must have the cGPU module installed to support GPU sharing and scheduling. This topic describes how to upgrade the cGPU module on a node using commands or the console.

Step 1: Upgrade components

Cluster type

Component upgrade method

ACK managed cluster Pro
ACK Edge cluster Pro

To upgrade the ack-ai-installer component, see Upgrade the shared GPU scheduling component.

ACK dedicated cluster

To upgrade the ack-cgpu component, perform the following steps:

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, find the cluster you want and click its name. In the left-side navigation pane, choose Applications > Helm.
On the Helm page, find the ack-cgpu component, click Update in the Actions column, select a Version, and then click OK.

Step 2: Upgrade existing nodes

Important

Stop the GPU applications on the node during the upgrade.
Upgrade one node first. After you verify that the GPU applications run as expected, upgrade other GPU nodes in batches.
This method resets the system disk of the node. If the system disk of your node contains data, first create a backup.

1. Remove and re-add the node

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of the one you want to change. In the left navigation pane, choose Nodes > Nodes.
On the Nodes page, select the cGPU node to upgrade and click Batch Remove. In the Remove Node dialog box, select Drain Node.
Re-add the removed node to the original node pool. For more information, see Add existing nodes to a cluster.
Important
Select the automatic node addition method. The node is not reset if you add it manually.

2. Verify the result

Run the following command to query the cgpu-installer that corresponds to the newly added node:

kubectl get po -l name=cgpu-installer -n kube-system -o wide

Expected output:

NAME                   READY   STATUS    RESTARTS   AGE    IP                NODE                         NOMINATED NODE   READINESS GATES
cgpu-installer-*****   1/1     Running   0          4d2h   192.168.XXX.XX1   cn-beijing.192.168.XXX.XX1   <none>           <none>
cgpu-installer-**2     1/1     Running   0          4d2h   192.168.XXX.XX2   cn-beijing.192.168.XXX.XX2   <none>           <none>
cgpu-installer-**3     1/1     Running   0          4d2h   192.168.XXX.XX3   cn-beijing.192.168.XXX.XX3   <none>           <none>

Run the following command to access the pod named cgpu-installer-******:
```
kubectl exec -ti cgpu-installer-xxxxx -n kube-system -- bash
```
Run the following command to query the current cGPU version:
```
nsenter -t 1 -i -p -n -u -m -- cat /proc/cgpu_km/version
```
Sample output:
```
1.5.16
```
Note
For information about the latest cGPU version, see ack-ai-installer.

cGPU version compatibility

NVIDIA driver compatibility

cGPU version

Compatible NVIDIA drivers

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.6

1.5.5

1.5.3

Supported:

460 series
470 series
510 series
515 series
525 series
535 series
550 series
560 series
565 series
570 series
575 series

1.5.2

1.0.10

1.0.9

1.0.8

1.0.7

1.0.6

1.0.5

Supported:

460 series
470 series <= 470.161.03
510 series <= 510.108.03
515 series <= 515.86.01
525 series <= 525.89.03

Not supported:

535 series
550 series
560 series
565 series
570 series
575 series

1.0.3

0.8.17

0.8.13

Supported:

460 series
470 series <= 470.161.03

Not supported:

510 series
515 series
525 series
535 series
550 series
560 series
565 series
570 series
575 series

Instance family compatibility

cGPU version

Compatible instance families

1.5.20

1.5.19

Supported:

gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e
gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e
gn8t / ebmgn8t
gn8is / gn8v / ebmgn8is / ebmgn8v
gn8ia / ebmgn8ia
ebmgn9t

1.5.18

1.5.17

1.5.16

1.5.15

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

Supported:

gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e
gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e
gn8t / ebmgn8t
gn8is / gn8v / ebmgn8is / ebmgn8v
gn8ia / ebmgn8ia

Not supported:

ebmgn9t

1.5.8

1.5.7

Supported:

gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e
gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e
gn8t / ebmgn8t
gn8is / gn8v / ebmgn8is / ebmgn8v

Not supported:

gn8ia / ebmgn8ia
ebmgn9t

1.5.6

1.5.5

Supported:

gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e
gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e
gn8t / ebmgn8t

Not supported:

gn8is / gn8v / ebmgn8is / ebmgn8v
gn8ia / ebmgn8ia
ebmgn9t

1.5.3

1.5.2

1.0.10

1.0.9

1.0.8

1.0.7

1.0.6

1.0.5

1.0.3

Supported:

gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e
gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e

Not supported:

gn8t / ebmgn8t
gn8is / gn8v / ebmgn8is / ebmgn8v
gn8ia / ebmgn8ia
ebmgn9t

0.8.17

0.8.13

Supported:

gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e

Not supported:

gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e
gn8t / ebmgn8t
gn8is / gn8v / ebmgn8is / ebmgn8v
gn8ia / ebmgn8ia
ebmgn9t

nvidia-container-toolkit compatibility

cGPU version

Compatible nvidia-container-toolkit

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.6

1.5.5

1.5.3

1.5.2

1.0.10

Supported:

nvidia-container-toolkit <= 1.10
nvidia-container-toolkit: 1.11 ~ 1.17

1.0.9

1.0.8

1.0.7

1.0.6

1.0.5

1.0.3

0.8.17

0.8.13

Supported:

nvidia-container-toolkit <= 1.10

Not supported:

nvidia-container-toolkit: 1.11 ~ 1.17

Kernel version compatibility

cGPU version

Compatible kernel versions

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

Supported:

kernel 3.x
kernel 4.x
kernel 5.x <= 5.15

1.5.8

1.5.7

1.5.6

1.5.5

1.5.3

Supported:

kernel 3.x
kernel 4.x
kernel 5.x <= 5.10

1.5.2

1.0.10

1.0.9

1.0.8

1.0.7

1.0.6

1.0.5

1.0.3

Supported:

kernel 3.x
kernel 4.x
kernel 5.x <= 5.1

0.8.17

Supported:

kernel 3.x
kernel 4.x
kernel 5.x <= 5.0

0.8.13

0.8.12

0.8.10

Supported:

kernel 3.x
kernel 4.x

Not supported:

kernel 5.x