Vulnerability CVE-2025-23266 - Container Service for Kubernetes

NVIDIA Container Toolkit 1.17.7 and earlier contain a time-of-check-to-time-of-use (TOCTOU) race condition when the default configuration is used. Successful exploitation can lead to a container escape that allows an attacker to run arbitrary commands or obtain sensitive information on the host. Fix affected GPU nodes promptly.

For more details about the vulnerability, see the NVIDIA Container Toolkit security bulletin. We recommend that you fix this vulnerability promptly.

Affected scope

This vulnerability affects clusters that run a Kubernetes version earlier than 1.32 and have NVIDIA Container Toolkit 1.17.7 or earlier installed on GPU nodes.

How to check the component version

You can run the following command to check the component version:

nvidia-container-cli --version

The following output is expected for version 1.17.8:

cli-version: 1.17.8
lib-version: 1.17.8
build date: 2025-05-30T13:47+00:00
build revision: 6eda4d76c8c5f8fc174e4abca83e513fb4dd63b0
build compiler: x86_64-linux-gnu-gcc-7 7.5.0
build platform: x86_64

Preventive measures

Until you apply the remediation, we recommend avoiding untrusted container images in your cluster to maintain security and stability.

Use the ACKAllowedRepos policy to restrict your cluster to using container images from trusted repositories only. For instructions, see Enable policy governance. As a best practice, also grant image import permissions only to trusted users, following the principle of least privilege.
Use Notation and Ratify to sign and verify OCI artifacts. This ensures that only trusted images can be deployed in the cluster.

Solutions

Notes

This fix applies only to the cloud node pools of ACK Pro clusters, ACK Basic clusters, ACK dedicated clusters, and ACK Edge clusters, and the managed node pools of ACK Lingjun clusters.
If your cluster is an ACK Lingjun cluster that uses Lingjun node pools, submit a ticket for assistance.
To ensure system stability, apply the fix to nodes in batches.
The remediation process restarts running application pods on the nodes. We recommend that you perform this operation during off-peak hours.

Fixes

New GPU nodes

This method applies only to clusters that run Kubernetes 1.20 or later. If your cluster runs an earlier version, upgrade your cluster.

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.
Find the corresponding node pool, click Edit, add the ack.aliyun.com/nvidia-container-runtime-version=1.17.8 label, and then click OK.
Note
- This label locks the nvidia-container-toolkit version to 1.17.8 for scaled-out nodes in the node pool. The version is not automatically updated when a new version of nvidia-container-toolkit is released.
- To use a new version of nvidia-container-toolkit, you must manually remove this label. Then, scaled-out nodes will use the latest version by default.

Existing GPU nodes

For existing GPU nodes, you can manually apply the fix by running a remediation script. The detailed steps are provided below.

Expand to view manual remediation methods

Prerequisites

Make sure that the Alibaba Cloud account or RAM user has the required permissions for CloudOps Orchestration Service (OOS), such as AliyunOOSFullAccess.
Check whether the cGPU add-on is installed in the cluster. If the cGPU add-on is not installed, proceed to Step 1: Drain the node. If the cGPU add-on is installed, make sure that the version of the add-on is 1.1.0 or later.
Expand to view how to check whether the cGPU add-on is installed and how to upgrade the add-on
1. Log on to the ACK console. In the left navigation pane, click Clusters.
2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Helm.
3. On the Helm page, check the component version.
  - If ack-cgpu does not exist, the cGPU add-on is not installed.
  - If both ack-ai-installer and ack-cgpu exist, submit a ticket for assistance.
  - If ack-ai-installer exists and its version is earlier than 1.7.5, upgrade the ack-ai-installer component. For more information, see Upgrade the shared GPU scheduling component.
  - If ack-cgpu exists and its version is earlier than 1.5.1, click Update to the right of the component and follow the on-screen instructions to upgrade the component.
4. After upgrading the related components, follow the instructions in Upgrade node cGPU version to upgrade the cGPU on existing nodes.

Step 1: Drain the node

Console

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Nodes.
On the Nodes page, select the checkbox of the target node, click Drain at the bottom of the page, and then click OK.

Kubectl

Run the following command to mark the node as unschedulable:
```
kubectl cordon <NODE_NAME>
```

Run the following command to drain the node:

kubectl drain <NODE_NAME> --grace-period=120 --ignore-daemonsets=true

Step 2: Run the remediation script

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Nodes.
On the Nodes page, select the target node, click Batch Operations at the bottom of the page, select Run Shell Scripts, and then click OK.
On the Basic Information tab of the CloudOps Orchestration Service (OOS) page, configure the parameters as described in the following table, and then click Next: Set Parameters.
Parameter
Value
Template Category
Public Template
Template
ACS-ECS-BulkyRunCommand
Execution Mode
Suspend upon Failure

On the Parameter Settings tab, select Run a Shell script, paste the following script into the Command Content field, and then click Next: Confirm.

#!/bin/bash
set -e
# Set the region where the node is located.
export REGION=$(curl 100.100.100.200/latest/meta-data/region-id 2>/dev/null)

if [[ $REGION == "" ]];then
    echo "Error: failed to get region"
    exit 1
fi

NV_TOOLKIT_VERSION=1.17.8

PACKAGE=upgrade_nvidia-container-toolkit-${NV_TOOLKIT_VERSION}.tar.gz

cd /tmp

curl -o ${PACKAGE}  http://aliacs-k8s-${REGION}.oss-${REGION}-internal.aliyuncs.com/public/pkg/nvidia-container-runtime/${PACKAGE} 

tar -xf ${PACKAGE} 

cd pkg/nvidia-container-runtime/upgrade/common

bash upgrade-nvidia-container-toolkit.sh

On the OK tab, verify the information and click Create.
After the task runs successfully, go to the CloudOps Orchestration Service console. In the navigation pane on the left, choose Automated Task > Task Execution Management. Find and click the relevant execution ID. Then, on the Details tab, view the Output.
- If the script returns the following output, the node was not vulnerable, and the script made no changes. You can ignore this message.
```
2025-03-22/xxxx  INFO  No need to upgrade current nvidia-container-toolkit(1.17.8)
```
- If the script returns the following output, the node was vulnerable and has been patched.
```
2025-03-22/xxxxx  INFO  succeed to upgrade nvidia container toolkit
```

Step 3: Make the node schedulable

Console

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Nodes.
On the Nodes page, select the target node, click Set Node Schedulability at the bottom of the page, select Set to Schedulable in the dialog box that appears, and then click OK.

kubectl

Run the following command to make the node schedulable:

kubectl uncordon <NODE_NAME>

(Optional) Step 4: Verify the GPU node

After completing the preceding steps, verify that the node works as expected by deploying a GPU application. Use a sample YAML file from one of the following topics.

exclusive GPU: Use the default Kubernetes GPU scheduling.
shared GPU: Run a shared GPU scheduling example.

Parameter	Value
Template Category	Public Template
Template	ACS-ECS-BulkyRunCommand
Execution Mode	Suspend upon Failure