:Configure GPU fault alerting and solutions

Background Information

1. Configure routine monitoring and alerting: You can configure alert rules based on GPU metrics (ACK GPU Monitoring 2.0) and events (ACK Node Problem Detector) based on your business requirements. When a fault occurs on a GPU, the system triggers an alert at the earliest opportunity. This improves the responsiveness of your system to GPU faults.

2. Container Intelligence Service (CIS)-assisted diagnostics and analytics: When a fault occurs on a GPU, the relevant GPU alerts or events may provide limited information for the GPU status and the details of the fault. In this case, you can use CIS to diagnose the node to which the faulty GPU belongs or the pod that uses the faulty GPU. CIS generates a diagnosis report that provides detailed information about the fault. The detailed information helps you identify the type of the fault and select suitable solutions to fix the fault.

3. Fault isolation and recovery: To mitigate the impact of specific faults, ACK allows you to enable node isolation and GPU isolation. After the fault is fixed, you can cancel the isolation to run your applications as normal.

Step 1: Configure routine monitoring and alerting

Node Problem Detector (NPD) supports regular GPU inspections and helps you detect GPU faults. You can specify contacts to receive alerts when NPD detects GPU faults.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, find the one you want to manage and click its name. In the left navigation pane, click Add-ons.

3. On the Add-ons page, search for ack-node-problem-detector and click Install after it appears.

   Note

   If you have previously installed the component, make sure that the version is 1.2.20 or later. For more information, see ack-node-problem-detector.

4. On the Clusters page, find the cluster you want and click its name. In the left navigation pane, choose Operations > Alerts.

5. On the Alerts page, click the Alert Contacts tab. Then, click Create and configure a contact based on the on-screen instructions.

6. On the Alert Rules tab, select Alert Rule Set for GPU Monitoring and click Status. Then, click Modify Contacts and select the contact you created.

Step 2: Use CIS to perform diagnostics and analysis

When faults occur on GPUs in a cluster, the cluster administrator is notified by text messages, emails, or DingTalk messages. You can log on to the ACK console and use CIS to locate and analyze the faults.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, find the cluster you want and click its name. In the left navigation pane, choose Inspections and Diagnostics > Diagnostics.

3. On the Diagnosis page, click Node diagnosis. In the upper-left corner of the Node diagnosis page, click Diagnosis.

4. In the Select node panel, specify the Node name parameter, read and select I know and agree, and then click Create diagnosis.

5. After the diagnosis is completed, the results of all diagnostic items are displayed. For more information about the diagnostic items supported by the node diagnostics feature, see Node diagnostics.

   If an Xid error occurs on a GPU, {"GPU-326dc0ce-XXXX-77b6-XXXX-9a2eeXXXX":["43"]} is displayed for the GPUXIDErrors item, which indicates that an Xid 43 error occurs on the GPU-xxx GPU. For more information about the GPU errors in the diagnosis reports provided by CIS, see Events of GPU faults detected by NPD.

Step 3: Isolate the faulty GPU

// Specify the index of the GPU.
{
    "index": ["1"]
}

// Specify the UUID of the GPU.
{
    "uuid": ["GPU-xxx-xxx-xxx"]
}

Events of GPU faults detected by NPD