What NVIDIA driver versions does ACK support? - Container Service for Kubernetes

Before installing an NVIDIA driver on a node, ensure the version is on the ACK-supported list. This topic lists supported NVIDIA driver versions.

CUDA

CUDA is a parallel computing platform and programming model from NVIDIA that uses GPUs to accelerate computing performance.

The following figure shows the CUDA architecture. The Driver API and Runtime API differ as follows:

Driver API: More feature-rich but more complex.
Runtime API: Wraps Driver API functions and hides initialization details for simpler usage.

The NVIDIA Driver package provides the CUDA Driver API, while the CUDA Toolkit package provides the CUDA Library and CUDA Runtime.

Driver and cluster version compatibility

Choosing and maintaining driver versions

ACK periodically updates the default driver version, so newly scaled-out GPU nodes may use a different version. To prevent this, specify a driver version for your node pool.
If the specified driver version is not listed in Driver and OS kernel version compatibility, ACK installs the default version. If you specify a version incompatible with the latest OS, node addition may fail. Select the latest supported driver version.
After an OS kernel upgrade, the GPU driver may become unavailable. Remove the node from the node pool and re-add it (which automatically adds the existing node back), or manually upgrade the GPU driver on a node.
If you customize the GPU driver version by specifying a version number or by using an OSS URL, the OS and driver may become incompatible after the update. See Supported NVIDIA driver versions in ACK to select a compatible driver.

Known issues in specific driver versions

Driver versions 510 and later may occasionally cause XID 119 or XID 120 errors. If this occurs, see Troubleshoot GPU failures caused by XID 119 or XID 120 errors.
Driver version 550 resolves issues such as frequent XID 119, 120, and 31 errors, and kernel panics in certain applications. We recommend upgrading existing GPU nodes to driver version 550.
When using a 570-series or later driver with the monitoring add-on, ensure ack-arms-prometheus is 1.1.33 or later and ack-gpu-exporter is 2.3.0 or later.

Driver requirements for specific instance types

For the gn9t instance family, do not use driver versions earlier than 570.153.02. Earlier versions may cause frequent GPU device disconnections. Symptoms:
- The nvidia-smi command reports fewer GPUs than physically installed, or outputs No devices were found.
- The lspci | grep -i nvidia command still detects the device, but its status shows [rev b0].
For L20N GPU instance types (ecs.ebmgn9g, ecs.ebmgn9gc, and ecs.ebmgn9ge), you must use GPU driver version 580 or later.

The following table lists NVIDIA GPU driver versions supported by each ACK cluster version.

Important

For ACK Lingjun clusters and Lingjun nodes in ACK managed Pro clusters, GPU drivers are built into the OS image. Installing a specific GPU driver version by node label is not supported. Edge node pools in ACK Edge clusters do not support installing a specific GPU driver version through node labels either.

Cluster version	Default driver version	Custom driver support	Supported driver versions
1.28 and later	535.161.07 570.169 (for ecs.gn9t and ecs.ebmgn9t instance types)	Yes	580.126.20 580.126.09 570.195.03 570.169 570.133.20 550.163.01 550.144.03 550.90.07 535.230.02 535.161.07 The following driver versions are incompatible with the latest operating systems. 535.129.03 525.147.05 515.105.01 510.108.03 535.54.03 525.105.17 515.86.01 510.47.03 470.161.03 470.103.01 470.82.01 470.57.02 460.91.03
1.26		Yes
1.24		Yes
1.22		Yes
1.20		Yes	450.119.04 450.102.04 450.82.02 450.51.06 418.181.07 418.87.01
1.18.8	418.181.07	Yes
1.16.9	418.181.07	Yes
1.16.6	418.87.01	No
1.14.8	418.181.07	Yes

Driver and GPU compatibility

Item	gn8v	gn8is	gn7e	gn7i	gn7	gn6e	gn6i	gn6v	gn5i	gn5
Product Type	Data Center / Tesla	Data Center / Tesla	Data Center / Tesla	Data Center / Tesla	Data Center / Tesla	Data Center / Tesla	Data Center / Tesla	Data Center / Tesla	Data Center / Tesla	Data Center / Tesla
Product series	H-Series	L-Series	A-Series	A-Series	A-Series	V-Series	T-Series	V-Series	P-Series	P-Series
Recommended Tesla driver version	570.133.20 or later		450.80.02 or later	460.73.01 or later	450.80.02 or later	410.79 or later
Recommended CUDA Toolkit version	CUDA Toolkit 12.4 Update 1		CUDA Toolkit 11.0 Update 1	CUDA Toolkit 11.2	CUDA Toolkit 11.0 Update 1	CUDA Toolkit 10.1 Update 2

Note

The preceding table lists GPU information for only some common GPU-accelerated compute-optimized instance types. Instances with the same GPU model share the same GPU information (product type, product series, and product family). For example, both ebmgn7i and gn7i instances use the NVIDIA A10 GPU. Therefore, these two instances have the same product type, product series, and product family.
When you manually install a Tesla driver and a CUDA package, ensure that the driver version is compatible with the CUDA package version. For more information, see CUDA Compatibility.

Driver and OS kernel version compatibility

See the kernel version and image ID mapping table below for kernel-to-image-ID mappings.

Driver version

Alibaba Cloud Linux 2

Alibaba Cloud Linux 3

CentOS 7

Ubuntu 22.04

580.126.20

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Unsupported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

580.126.09

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Unsupported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

570.195.03

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Unsupported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

570.169

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Unsupported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

570.133.20

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Unsupported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

550.163.01

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

550.144.03

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

550.90.07

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

550.54.15

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

550.54.14

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

535.247.01

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

535.230.02

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

535.161.07

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

535.129.03

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64]

Unsupported range:

[5.10.134-18.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

535.98

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64]

Unsupported range:

[5.10.134-18.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

535.54.03

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64]

Unsupported range:

[5.10.134-18.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

525.147.05

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64]

Unsupported range:

[5.10.134-18.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

525.105.17

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64]

Unsupported range:

[5.10.134-18.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

515.105.01

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64]

Unsupported range:

[5.10.134-18.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

515.86.01

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64]

Unsupported range:

[5.10.134-18.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

510.108.03

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64]

Unsupported range:

[5.10.134-18.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

510.54

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64]

Unsupported range:

[5.10.134-15.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

510.47.03

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64]

Unsupported range:

[5.10.134-15.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

470.256.02

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, ∞)

470.161.03

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-17.3.al8.x86_64]

Unsupported range:

[5.10.134-18.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

470.103.01

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64]

Unsupported range:

[5.10.134-15.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

470.82.01

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64]

Unsupported range:

[5.10.134-15.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

470.57.02

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64]

Unsupported range:

[5.10.134-15.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

460.106.00

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64]

Unsupported range:

[5.10.134-15.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Supported range:

[5.15.0-40-generic, 5.15.0-101-generic]

Unsupported range:

[5.15.0-106-generic, ∞)

460.91.03

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64]

Unsupported range:

[5.10.134-15.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

460.73.01

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64]

Unsupported range:

[5.10.134-15.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

460.32.03

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64]

Unsupported range:

[5.10.134-15.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

450.119.04

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64]

Unsupported range:

[5.10.134-15.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

450.102.04

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Supported range:

[5.10.23-5.al8.x86_64, 5.10.134-14.al8.x86_64]

Unsupported range:

[5.10.134-15.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

450.80.02

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Unsupported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

440.33.01

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Unsupported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

418.181.07

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Unsupported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

418.113

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Unsupported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

418.87.01

Supported range:

[4.19.81-17.1.al7.x86_64, ∞)

Unsupported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

410.93

Supported range:

[4.19.81-17.1.al7.x86_64, 4.19.91-18.al7.x86_64]

Unsupported range:

[4.19.91-19.1.al7.x86_64, ∞)

Unsupported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, 3.10.0-957.21.3.el7.x86_64]

Unsupported range:

[3.10.0-1062.9.1.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

410.79

Supported range:

[4.19.81-17.1.al7.x86_64, 4.19.91-18.al7.x86_64]

Unsupported range:

[4.19.91-19.1.al7.x86_64, ∞)

Unsupported range:

[5.10.23-5.al8.x86_64, ∞)

Supported range:

[3.10.0-862.14.4.el7.x86_64, 3.10.0-957.21.3.el7.x86_64]

Unsupported range:

[3.10.0-1062.9.1.el7.x86_64, ∞)

Unsupported range:

[5.15.0-40-generic, ∞)

Kernel version and image ID mapping

Kernel version	Image ID
5.15.0-106-generic	ubuntu_22_04_x64_20G_alibase_20240508.vhd
5.15.0-101-generic	ubuntu_22_04_x64_20G_alibase_20240322.vhd
5.15.0-40-generic	ubuntu_22_04_x64_20G_alibase_20220628.vhd
5.10.134-18.al8.x86_64	aliyun_3_x64_20G_container_optimized_20250117.vhd
5.10.134-17.3.al8.x86_64	aliyun_3_x64_20G_alibase_20241103.vhd
5.10.134-15.al8.x86_64	aliyun_3_x64_20G_alibase_20230727.vhd
5.10.134-14.al8.x86_64	aliyun_3_x64_20G_alibase_20230516.vhd
5.10.23-5.al8.x86_64	aliyun_3_x64_20G_alibase_20210425.vhd
4.19.91-19.1.al7.x86_64	aliyun_2_1903_x64_20G_alibase_20200529.vhd
4.19.91-18.al7.x86_64	aliyun_2_1903_x64_20G_alibase_20200324.vhd
4.19.81-17.1.al7.x86_64	aliyun_2_1903_x64_20G_alibase_20200221.vhd
3.10.0-1062.9.1.el7.x86_64	centos_7_7_x64_20G_alibase_20191225.vhd
3.10.0-957.21.3.el7.x86_64	centos_7_6_x64_20G_alibase_20211130.vhd
3.10.0-862.14.4.el7.x86_64	centos_7_5_x64_20G_alibase_20211130.vhd

Driver and CUDA Toolkit compatibility

Select a driver version compatible with your application's CUDA Toolkit version. See CUDA Toolkit Release Notes for the compatibility matrix.

Checking CUDA Driver API version

Run the nvidia-smi command on a node to check the installed driver version and CUDA Driver API version. In the following output, 550.144.03 is the driver version, and CUDA version 12.6 indicates the driver supports the CUDA Runtime API up to version 12.6.

Mon Mar 24 08:51:55 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P4                       On  |   00000000:00:07.0 Off |                    0 |
| N/A   33C    P8              7W /   75W |       0MiB /   7680MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Runtime API version

NVIDIA provides official CUDA base images with the CUDA Toolkit pre-installed. Build your application image on a base image matching the toolkit version your application requires.

The CUDA base image of your Docker image determines the CUDA Runtime API version available to your application. For example, building on NVIDIA/CUDA:12.2.0-base-Ubuntu20.04 means the application uses CUDA Runtime API version 12.2.0.

Container Service for Kubernetes:ACK-supported NVIDIA driver versions

CUDA

Driver and cluster version compatibility

Driver and GPU compatibility

Driver and OS kernel version compatibility

Driver and CUDA Toolkit compatibility

Checking CUDA Driver API version

Runtime API version

References