ACK uses the ack-node-problem-detector (ACK NPD) component to monitor GPU health. When a GPU node encounters an exception, such as an XID or SXID error, ACK NPD automatically detects the faulty GPU and works with the ACK NVIDIA Device Plugin component to fence it. This process keeps healthy GPUs available to serve workloads, minimizing business impact and improving cluster reliability and efficiency.
ack-node-problem-detector (ACK NPD) is a monitoring component for cluster node anomalies, adapted and enhanced by ACK from the open-source node-problem-detector project. It provides a rich set of checks for GPU-specific anomalies to enhance fault detection for GPU workloads. When the component detects an anomaly, it generates a Kubernetes Event and Node Condition based on the anomaly type.
Usage notes
If a faulty GPU card is isolated, tasks may fail to be scheduled if the remaining GPUs on the node cannot meet task requirements. For example, a task that requires eight GPU cards cannot run if only seven are available. This can leave GPU resources idle. Automatic GPU isolation is not the same as automatic repair, and billing continues for the node instance. You must still repair the node. We recommend configuring GPU exception alerts to ensure timely remediation. The isolation is lifted automatically once the GPU status returns to normal.
You can also enable or disable automatic GPU isolation based on your business requirements. Specific versions of the NVIDIA Device Plugin component support automatic isolation for faulty GPU cards, but the procedure for disabling this capability varies. For detailed instructions, see How to disable the native GPU isolation capability of the NVIDIA Device Plugin.
NVIDIA's XIDs and SXIDs are written by the GPU driver to
/var/log/messagesor/var/log/syslogthrough the NVRM event mechanism. ACK NPD tracks whether each XID and SXID has been processed. If a node is rebooted after an XID or SXID is detected, ACK NPD considers the issue resolved and will not generate an Event or a Node Condition for it. This occurs even if the underlying problem is not fixed. For example, an XID 79 error requires a GPU device replacement to be resolved, but ACK NPD will still mark the error as resolved after a node reboot.ACK NPD detects NVIDIA XID or NVIDIA SXID by checking the
/var/log/messagesor/var/log/syslogfile on the node. If the dmesg log is redirected to another file, ACK NPD cannot detect NVIDIA XID and SXID.As of ACK NPD version 1.2.29, ACK NPD's GPU fault detection plug-in is deployed as a separate DaemonSet named ack-accel-health-monitor.
In some cases, a GPU fault on a node may prevent new GPU containers from being created. This can cause the GPU fault detection container itself to fail to start, which prevents the detection process from running correctly.
The ACK NPD GPU detection plugin Pod requires high privileges, such as
privileged=true, to detect the status of GPU devices and GPU components. For details, see the table below.Cluster RBAC permissions
Container permissions
Node: get
Node/Status: update
Events: create
privileged: trueMounts the host
/dev/kmsgas read-only.Mount the host's
/usr/libas read-onlyMount the host's
/etcdirectory as read-only.Read-only mount of the host's
/usr/lib64Mount the host
/procas read-only
GPU automatic isolation
Starting with version 1.2.35 of the ACK Node Problem Detector (ACK NPD) component and version 0.7.0 of the ACK NVIDIA Device Plugin component, the automatic GPU isolation mechanism has shifted from a default trigger to a configuration-based trigger. For more information, see [Product Change] Announcement on Changes to the Automatic GPU Isolation Feature.
Legacy mechanism (default trigger)
When the ACK Node Problem Detector component detects a GPU fault, it generates a GPU isolation file. The ACK NVIDIA Device Plugin component then isolates all GPU devices listed in this file. By default, the system automatically isolates GPUs after detecting specific faults. You can enable or disable this feature by configuring whether to generate the GPU isolation file.
New mechanism (configuration-based trigger)
When the ACK Node Problem Detector component detects a GPU fault, it generates a fault detection report. The ACK NVIDIA Device Plugin component determines whether to isolate the GPU based on this report and the NPD check items that you configure as isolation triggers. By default, no trigger check items are configured for the ACK NVIDIA Device Plugin component, so automatic GPU isolation is disabled by default. You can configure these check items to specify which faults trigger automatic GPU isolation.
For the applicability of each mechanism, see GPU automatic isolation behavior.
To maintain compatibility, the new version of ACK NPD will continue to generate the old-format GPU isolation file. However, the new version of ACK NVIDIA Device Plugin no longer reads this file. Isolation behavior is determined entirely by its own configuration.
GPU automatic isolation behavior
This new mechanism applies only to ACK clusters running Kubernetes version 1.32 and later.
Clusters on Kubernetes versions below 1.32 will continue to use the existing isolation mechanism.
The behavior of the automatic GPU isolation feature varies based on the combination of add-on versions:
ACK NPD version | ACK NVIDIA Device Plugin version | GPU automatic isolation behavior | Actions |
ACK NPD version < 1.2.24 | N/A | GPU anomaly detection is not supported. | N/A |
ACK NPD version ≥ 1.2.24 | ACK NVIDIA Device Plugin version < 0.7.0 | Isolation follows the legacy mechanism. | Follow the legacy procedure. |
1.2.24 ≤ ACK NPD version < 1.2.35 | ACK NVIDIA Device Plugin version ≥ 0.7.0 | Automatic GPU isolation does not take effect, but other features work as expected. Earlier ACK NPD versions do not generate anomaly detection reports. Consequently, the new ACK NVIDIA Device Plugin cannot identify faulty GPUs and does not perform automatic isolation. | N/A Upgrade ACK NPD to the latest version to use the new mechanism. |
ACK NPD version ≥ 1.2.35 This version is in canary release. Submit a ticket for allowlist access. | ACK NVIDIA Device Plugin version ≥ 0.7.0 This version is in canary release. Submit a ticket for allowlist access. | Isolation follows the new mechanism. | Follow the new procedure. |
For instructions on checking and upgrading the ACK NPD and ACK NVIDIA Device Plugin components, see Check or upgrade the ACK NPD version and Check or upgrade the NVIDIA Device Plugin version.
Enable or disable GPU automatic isolation
If you disable GPU automatic isolation or if the feature is not working, only the automatic isolation of GPUs is affected. When ACK NPD detects a GPU exception, it still triggers Node Conditions, Kubernetes Events, and alarms based on the ACK NPD check items, but it does not automatically isolate the faulty GPU.
New method
When enabled, a faulty GPU is automatically isolated if it triggers a configured ACK NPD check item. Automatic isolation is not the same as automatic repair. The node instance with the isolated GPU continues to be billed. You must still repair the node. We recommend configuring GPU exception alarms for prompt handling. If no check items are selected, automatic isolation is not triggered.
Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Add-ons.
On the component management page, search for the ack-nvidia-device-plugin component and click Configuration on its card.
In the dialog box that appears, enable or disable GPU automatic isolation.
Enable: Select the Enable GPU Automatic Isolation checkbox, and then select the NPD check items to trigger GPU automatic isolation.
Disable: Clear the Enable GPU Automatic Isolation checkbox, or do not select any check items.
Legacy method
When a GPU exception is detected, the ack-node-problem-detector component generates an NVIDIA GPU isolation file based on the default isolation policy. The ack-nvidia-device-plugin component then automatically isolates the faulty GPU based on the file's content. This prevents new workloads from being scheduled to the faulty GPU, while other healthy GPUs on the node remain available to serve workloads. However, automatic isolation does not perform automatic repair. The node instance with the isolated GPU continues to be billed. You must still restart or repair the node manually. We recommend that you configure GPU exception alarms for prompt handling.
For ack-node-problem-detector v1.2.30 or later, use the
generateNvidiaGpuIsolationFilesetting in component management to control the automatic isolation of faulty GPUs.After you upgrade the component, see GPU automatic isolation behavior to determine how to enable or disable GPU automatic isolation for the new component version.
Enable: In the legacy mechanism, GPU automatic isolation is enabled by default. You can re-enable it by setting
generateNvidiaGpuIsolationFile(for v1.2.30 or later) orEnabledIsolateGPU(for v1.2.24 to v1.2.29) totrue.Disable:
On the Clusters page, click the name of your cluster. In the left navigation pane, click Components and Add-ons.
On the Logs and Monitoring tab, find the ack-node-problem-detector component and follow the steps for your component version:
v1.2.24 to v1.2.29: Upgrade to the latest version.
v1.2.30 and later: Click Configuration.
On the component upgrade or configuration page, set the
generateNvidiaGpuIsolationFileparameter tofalseand click OK.NoteIf you previously disabled GPU automatic isolation in versions 1.2.24 to 1.2.29 by modifying the
ack-node-problem-detector-daemonsetand setting theEnabledIsolateGPUparameter tofalse, this setting is automatically retained when you upgrade the ACK NPD component. If you want to re-enable GPU automatic isolation, setgenerateNvidiaGpuIsolationFiletotrue.
To undo an existing GPU isolation, log on to the node where the XID error occurred and delete the
/etc/nvidia-device-plugin/unhealthyDevices.jsonfile. To prevent the GPU from being isolated again, disable the feature as described previously.
Detections and remediation
If a GPU exception is detected, see Nvidia Xid Errors for solutions. You can also check the console of the corresponding cloud product (such as ECS or Lingjun) for related O&M events, or use a self-diagnosis tool to troubleshoot hardware exceptions.
In the following table, Auto-isolate Faulty GPU is the default isolation behavior of the legacy mechanism. In the new mechanism, GPUs are isolated based on your custom configuration. For details, see automatic GPU isolation mechanism.
Other related events
In dedicated GPU scenarios, the legacy mechanism automatically isolates faulty GPUs, while the new mechanism requires you to configure triggers in component management. After isolation, new GPU application Pods are not scheduled on the GPU card. To verify the isolation, check the number of nvidia.com/gpu resources reported by the Kubernetes Node. After the GPU card recovers, ACK automatically releases it from isolation.
Trigger reason | Event content | Description |
GPU isolation | Yes
| The system isolates the GPU card after detecting an issue. |
GPU release from isolation | Yes
| The GPU card recovers, and ACK releases it from isolation. |