When you install an NVIDIA driver for a node, you must use a driver version that is supported by ACK. This topic lists the NVIDIA driver versions supported by ACK.
Introduction to CUDA
CUDA is a parallel computing platform and programming model introduced by NVIDIA in 2007. CUDA uses a graphics processing unit (GPU) to significantly improve computing performance.
The following figure shows the CUDA architecture. The CUDA software stack includes a driver layer API and a runtime layer API, which differ in the following ways:
Driver API: This API is feature-rich but complex to use.
CUDA Runtime API: This API encapsulates some driver APIs. It hides some driver initialization operations and is easier to use.
The CUDA Driver API is provided by the NVIDIA Driver package. The CUDA Library and CUDA Runtime are provided by the CUDA Toolkit package.

Driver and cluster version compatibility
The following table lists the NVIDIA GPU driver versions supported by different ACK cluster versions.
For ACK Lingjun clusters and Node Lingjun in ACK Pro clusters, the GPU driver is built into the operating system (OS) image. You cannot use node labels to install a specific GPU driver version. Similarly, for edge node pools in ACK Edge clusters, you cannot use node labels to install a specific GPU driver version.
Driver versions 510 and later may occasionally cause XID 119 or XID 120 errors. If you encounter these errors, see What do I do if an XID 119 or XID 120 error occurs and causes a GPU card to become unavailable when I use a GPU? for troubleshooting information.
Driver version 550 fixes issues for some applications, such as frequent XID 119, 120, or 31 errors and kernel panics. Upgrade your existing GPU nodes to driver version 550.
ACK periodically updates the default driver version for different cluster versions. This may cause the driver version on newly added GPU nodes in your cluster to change. To prevent this, you can specify a driver version for the node pool in your cluster. For more information, see Customize the GPU driver version for a node by specifying a version number.
When you create a node pool, if the driver version you specify is not listed in Driver and operating system kernel version compatibility, ACK automatically installs the default driver version. If you specify a driver version that is incompatible with the latest operating system, the node may fail to join the cluster. In this case, you must select the latest supported driver version.
After you upgrade the OS kernel version, the GPU driver installed on the node may become unavailable. To resolve this issue, you can remove the node from the node pool and then add it back, or manually upgrade the GPU driver of a node.
When you use monitoring components with driver versions 570 or later, the component versions must be ack-arms-prometheus 1.1.33 or later and ack-gpu-exporter 2.3.0 or later.
If your node pool uses custom GPU drivers configured by specifying driver versions or adding OSS URLs, OS image upgrades may trigger OS-driver incompatibility issues. Always select the latest validated driver version from the List of NVIDIA driver versions supported by ACK.
Cluster version | Default driver version | Supports custom driver versions | Supported NVIDIA driver versions |
1.28 and later | 535.161.07 570.169 (for ecs.gn9t and ecs.ebmgn9t instance types) | Yes |
The following driver versions are incompatible with the latest operating systems.
|
1.26 | Yes | ||
1.24 | Yes | ||
1.22 | Yes | ||
1.20 | Yes |
| |
1.18.8 | 418.181.07 | Yes | |
1.16.9 | 418.181.07 | Yes | |
1.16.6 | 418.87.01 | No | |
1.14.8 | 418.181.07 | Yes |
Driver, GPU card, and instance type compatibility
Driver and operating system kernel version compatibility
For more information about the mapping between kernel versions and OS image IDs, see Mapping between kernel versions and image IDs.
Driver version | Alibaba Cloud Linux 2 | Alibaba Cloud Linux 3 | CentOS | Ubuntu |
570.169 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, ∞) | Unsupported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, ∞) |
570.133.20 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, ∞) | Unsupported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, ∞) |
550.163.01 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, ∞) |
550.144.03 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, ∞) |
550.90.07 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, ∞) |
550.54.15 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, ∞) |
550.54.14 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, ∞) |
535.247.01 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, ∞) |
535.230.02 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, ∞) |
535.161.07 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, ∞) |
535.129.03 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
535.98 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
535.54.03 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
525.147.05 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
525.105.17 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
515.105.01 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
515.86.01 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
510.108.03 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
510.54 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
510.47.03 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
470.256.02 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, ∞) |
470.161.03 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64] Unsupported range: [5.10.134-18.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
470.103.01 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
470.82.01 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
470.57.02 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
460.106.00 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Supported range: [5.15.0-40-generic, 5.15.0-101-generic] Unsupported range: [5.15.0-106-generic, ∞) |
460.91.03 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
460.73.01 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
460.32.03 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
450.119.04 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
450.102.04 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Supported range: [5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64] Unsupported range: [5.10.134-15.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
450.80.02 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Unsupported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
440.33.01 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Unsupported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
418.181.07 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Unsupported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
418.113 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Unsupported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
418.87.01 | Supported range: [4.19.81-17.1.al7.x86_64, ∞) | Unsupported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
410.93 | Supported range: [4.19.81-17.1.al7.x86_64, 4.19.91-18.al7.x86_64] Unsupported range: [4.19.91-19.1.al7.x86_64, ∞) | Unsupported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, 3.10.0-957.21.3.el7.x86_64] Unsupported range: [3.10.0-1062.9.1.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
410.79 | Supported range: [4.19.81-17.1.al7.x86_64, 4.19.91-18.al7.x86_64] Unsupported range: [4.19.91-19.1.al7.x86_64, ∞) | Unsupported range: [5.10.23-5.al8.x86_64, ∞) | Supported range: [3.10.0-862.14.4.el7.x86_64, 3.10.0-957.21.3.el7.x86_64] Unsupported range: [3.10.0-1062.9.1.el7.x86_64, ∞) | Unsupported range: [5.15.0-40-generic, ∞) |
Driver and CUDA Toolkit compatibility
You can select an appropriate NVIDIA driver version based on the CUDA Toolkit version that your application uses and the compatibility list of CUDA Toolkit and NVIDIA driver versions. For more information, see cuda-toolkit-release-notes.
Obtain the driver layer API version
If an NVIDIA driver package is installed on a node, you can run the nvidia-smi command to view the driver version and the CUDA driver API version. In the following example, the installed driver version is 550.144.03 and the driver API version is 12.6. This indicates that the driver supports CUDA runtime API versions up to 12.6.
Mon Mar 24 08:51:55 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla P4 On | 00000000:00:07.0 Off | 0 |
| N/A 33C P8 7W / 75W | 0MiB / 7680MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+Obtain the runtime API version
When you install CUDA Toolkit in a container image, you can use the CUDA base images provided by NVIDIA. CUDA Toolkit is pre-installed in these base images. You can build your application container images based on these base images. You can also select different CUDA base images based on the CUDA Toolkit version.
When you use GPUs in containers, the CUDA runtime API version used by your application is determined by the CUDA base image of the application's Docker image. For example, if your application's Docker image is built based on the NVIDIA/CUDA:12.2.0-base-Ubuntu20.04 CUDA base image, the CUDA runtime API version used by the application is 12.2.0.