NVIDIA Container Toolkit 1.16.1 and earlier contains a Time-of-check to Time-of-Use (TOCTOU) vulnerability that can lead to container escape, allowing attackers to execute arbitrary commands on the host or access sensitive information. This vulnerability only applies when the toolkit is used with its default configuration.
This vulnerability does not affect Container Device Interface (CDI) usage. If your nodes use CDI, no action is required.
For details, see the NVIDIA security bulletin.
Patch nodes in batches to maintain cluster stability — draining nodes sequentially ensures PodDisruptionBudgets are respected and workloads remain available. Do not patch all nodes at the same time.
Scope of impact
GPU-accelerated nodes in ACK Edge clusters running NVIDIA Container Toolkit 1.16.1 or earlier are affected.
| Component | Affected versions | Fixed version |
|---|---|---|
| NVIDIA Container Toolkit | 1.16.1 and earlier | 1.16.2 |
Solutions
New nodes (created on or after October 27, 2024): ACK Edge has already deployed NVIDIA Container Toolkit 1.16.2 on clusters running Kubernetes 1.20 or later, so these nodes are not affected. If your cluster runs an earlier Kubernetes version, update the cluster version to get the fix.
Existing nodes (created before October 27, 2024): Run the CVE repair script manually. For cloud nodes, see cloud node repair instructions. For edge nodes, follow the steps below.
Step 1: Drain a node
Drain the node to evict running workloads before applying the patch. Select one of the following methods.
ACK console
-
Log on to the ACK console. In the left-side navigation pane, click Clusters.
-
On the Clusters page, click the name of the target cluster. In the left-side navigation pane, choose Nodes > Nodes.
-
Select the target node, click Drain at the bottom of the page, then click OK.
kubectl
-
Set the node to unschedulable:
kubectl cordon <NODE_NAME> -
Drain the node:
kubectl drain <NODE_NAME> --grace-period=120 --ignore-daemonsets=true
Step 2: Run the repair script
Run the following script on the affected node:
export REGION="" INTERCONNECT_MODE=""; export INTERNAL=$( [ "$INTERCONNECT_MODE" = "private" ] && echo "-internal" || echo "" ); wget http://aliacs-k8s-${REGION}.oss-${REGION}${INTERNAL}.aliyuncs.com/public/pkg/edge/fix-nvidia-cve.sh -O /tmp/fix-nvidia-cve.sh && bash /tmp/fix-nvidia-cve.sh;Replace the following parameters before running:
| Parameter | Description | Example |
|---|---|---|
REGION |
Region ID of the cluster. For supported regions, see Supported regions. | cn-hangzhou |
INTERCONNECT_MODE |
Network type for connecting to the node: basic for public network, private for Express Connect circuits. |
basic |
Verify the output:
-
If the node is not affected, the script prints:
The current version of Nvidia container toolkit is safe, no cve. -
If the vulnerability was present and is now fixed, the script prints:
2024-10-10/xxxxx INFO succeeded to fix nvidia container toolkit cve
Step 3: Bring the node online
After the script completes, restore the node to schedulable. Select one of the following methods.
ACK console
-
Log on to the ACK console. In the left-side navigation pane, click Clusters.
-
On the Clusters page, click the name of the target cluster. In the left-side navigation pane, choose Nodes > Nodes.
-
Select the target node, click Set Node Schedulability at the bottom of the page, select Set to Schedulable, then click OK.
kubectl
kubectl uncordon <NODE_NAME>Security hardening
After patching, enable security policy management and apply the ACKAllowedRepos policy to restrict image pulls to trusted repositories and enforce the principle of least privilege for image imports.