ACK supports a fixed set of NVIDIA driver versions for GPU nodes. Install only a driver version listed in this topic on GPU nodes in your cluster.
Driver and cluster version compatibility
The following driver versions are incompatible with the latest operating systems. Do not use them on new nodes: 535.129.03, 525.147.05, 515.105.01, 510.108.03, 535.54.03, 525.105.17, 515.86.01, 510.47.03, 470.161.03, 470.103.01, 470.82.01, 470.57.02, 460.91.03.
Usage notes:
-
GPU drivers are pre-installed in the OS images of ACK Lingjun clusters and Lingjun nodes in ACK managed Pro clusters. Installing specific driver versions via node labels is not supported for these nodes. Edge node pools in ACK Edge clusters have the same restriction.
-
Driver versions 510 and later may occasionally trigger XID 119 or XID 120 errors. For troubleshooting steps, see How to troubleshoot GPU disconnection caused by XID 119/XID 120 errors?
-
Driver version 550 fixes frequent XID 119, XID 120, XID 31 errors, and kernel panic issues in certain applications. Upgrade existing GPU nodes to driver version 550.
-
ACK periodically updates the default driver version. Newly created GPU nodes may use a different driver version than existing nodes. To prevent this, specify a driver version for your node pool.
-
When creating a node pool, if the specified driver version is not listed in Driver and OS kernel version compatibility, ACK installs the default driver version. If the specified version is incompatible with the latest OS, the node may fail to join. In that case, select the latest supported driver version.
-
After an OS kernel upgrade, the GPU driver on the node may become unavailable. To resolve this, remove the node from its node pool and re-add it, or manually upgrade the GPU node driver.
-
For driver series 570 or later, the minimum required add-on versions are: ack-arms-prometheus >= 1.1.33 and ack-gpu-exporter >= 2.3.0.
-
For the
gn9tinstance family, do not use driver versions earlier than 570.153.02. Earlier versions may cause GPU device disconnection with the following symptoms:-
nvidia-smireports fewer GPUs than physically present, or outputsNo devices were found. -
lspci | grep -i nvidiastill detects the device, but the device status shows[rev b0].
-
-
If you specify a driver version by version number or use an OSS URL, the OS and driver may become incompatible after an OS upgrade. Check this page and select the latest compatible driver.
|
Cluster version |
Default driver version |
Supports custom driver version? |
Supported NVIDIA driver versions |
|
1.28 and later |
535.161.07 570.169 (for ecs.gn9t and ecs.ebmgn9t instances) |
Yes |
The following driver versions are incompatible with the latest operating systems.
|
|
1.26 |
Yes |
||
|
1.24 |
Yes |
||
|
1.22 |
Yes |
||
|
1.20 |
Yes |
|
|
|
1.18.8 |
418.181.07 |
Yes |
|
|
1.16.9 |
418.181.07 |
Yes |
|
|
1.16.6 |
418.87.01 |
No |
|
|
1.14.8 |
418.181.07 |
Yes |
Driver and GPU instance type compatibility
The table below covers common GPU-accelerated compute-optimized instance types. Instances with the same GPU model share the same product type, series, and family — for example, both ebmgn7i and gn7i use the NVIDIA A10 GPU.
When manually installing a Tesla driver and a CUDA package, verify that the versions are compatible. See CUDA Compatibility.
|
Instance type |
gn8v |
gn8is |
gn7e |
gn7i |
gn7 |
gn6e |
gn6i |
gn6v |
gn5i |
gn5 |
|
Product type |
Data Center / Tesla |
Data Center / Tesla |
Data Center / Tesla |
Data Center / Tesla |
Data Center / Tesla |
Data Center / Tesla |
Data Center / Tesla |
Data Center / Tesla |
Data Center / Tesla |
Data Center / Tesla |
|
Product series |
H-Series |
L-Series |
A-Series |
A-Series |
A-Series |
V-Series |
T-Series |
V-Series |
P-Series |
P-Series |
|
Recommended Tesla driver version |
Version 570.133.20 or later |
Version 450.80.02 or later |
Version 460.73.01 or later |
Version 450.80.02 or later |
Version 410.79 or later |
|||||
|
Recommended CUDA Toolkit version |
||||||||||
Driver and OS kernel version compatibility
For the mapping between kernel versions and OS image IDs, see the Kernel version and image ID mapping table.
|
Driver version |
Alibaba Cloud Linux 2 |
Alibaba Cloud Linux 3 |
CentOS |
Ubuntu |
|
570.195.03 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, ∞) |
Unsupported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, ∞) |
|
570.169 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, ∞) |
Unsupported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, ∞) |
|
570.133.20 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, ∞) |
Unsupported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, ∞) |
|
550.163.01 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, ∞) |
|
550.144.03 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, ∞) |
|
550.90.07 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, ∞) |
|
550.54.15 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, ∞) |
|
550.54.14 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, ∞) |
|
535.247.01 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, ∞) |
|
535.230.02 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, ∞) |
|
535.161.07 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, ∞) |
|
535.129.03 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
535.98 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
535.54.03 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
525.147.05 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
525.105.17 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
515.105.01 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
515.86.01 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
510.108.03 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
510.54 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
510.47.03 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
470.256.02 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, ∞) |
|
470.161.03 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
470.103.01 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
470.82.01 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
470.57.02 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
|
460.106.00 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
|
460.91.03 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
|
460.73.01 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
|
460.32.03 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
|
450.119.04 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
|
450.102.04 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
|
450.80.02 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Unsupported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
|
440.33.01 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Unsupported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
|
418.181.07 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Unsupported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
|
418.113 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Unsupported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
|
418.87.01 |
Supported range: [4.19.81-17.1.al7.x86_64, ∞) |
Unsupported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
|
410.93 |
Supported range: [4.19.81-17.1.al7.x86_64, 4.19.91-18.al7.x86_64] Unsupported range: [4.19.91-19.1.al7.x86_64, ∞) |
Unsupported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, 3.10.0-957.21.3.el7.x86_64] Unsupported range: [3.10.0-1062.9.1.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
|
410.79 |
Supported range: [4.19.81-17.1.al7.x86_64, 4.19.91-18.al7.x86_64] Unsupported range: [4.19.91-19.1.al7.x86_64, ∞) |
Unsupported range: [5.10.23-5.al8.x86_64, ∞) |
Supported range: [3.10.0-862.14.4.el7.x86_64, 3.10.0-957.21.3.el7.x86_64] Unsupported range: [3.10.0-1062.9.1.el7.x86_64, ∞) |
Unsupported range: [5.15.0-40-generic, ∞) |
Driver and CUDA Toolkit compatibility
Select an NVIDIA driver version that is compatible with your application's CUDA Toolkit version. For the CUDA Toolkit and driver compatibility matrix, see CUDA Toolkit release notes.
Check driver and CUDA API versions on a node
Run nvidia-smi on a node that has a driver installed to view the driver version and the CUDA Driver API version it supports.
Mon Mar 24 08:51:55 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla P4 On | 00000000:00:07.0 Off | 0 |
| N/A 33C P8 7W / 75W | 0MiB / 7680MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
In this example, driver version 550.144.03 is installed and supports the CUDA Runtime API up to version 12.6.
Check the CUDA Runtime API version in a container
In GPU-enabled containers, the CUDA Runtime API version is determined by the CUDA base image used to build your application's container image — not by the node driver version.
Start with the official CUDA base images from NVIDIA, which have the CUDA Toolkit pre-installed. Choose a base image that matches your required CUDA Toolkit version. For example, if your container image is built from nvidia/cuda:12.2.0-base-ubuntu20.04, the application uses CUDA Runtime API version 12.2.0.