:End-to-end GPU fault diagnostics and recovery

Introduction

To quickly identify and fix GPU faults, an end-to-end fault diagnostics workflow is needed.

1. Triggers of fault diagnostics: Identify the source that triggers fault diagnostics.

2. Fault diagnostics: Check whether faults exist, observe the symptoms, and identify the cause by using logs, monitoring systems, and other diagnostic tools.

3. Fault isolation: Isolate the faulty component from the workflow in case the fault spreads.

4. Fault confirmation: After the diagnostic, confirm the occurrence of the fault again and take appropriate measures.

5. Fault recovery: Fix the fault based on the cause identified in the previous step.

6. Fault isolation removal: After the issue is resolved, redeploy the repaired resources in the production environment and restore the business.

The following figure shows the entire workflow.

Step 1: Triggers of fault diagnostics

Triggers of fault diagnostics include Kubernetes events, Managed Service for Prometheus, routine inspections, manually triggered diagnostics, ECS events, applications, and application controllers.

Step 2: Fault diagnostics

Fault diagnostic is an important component of the fault diagnostics and recovery workflow. The following figure shows the fault diagnostics workflow in an ACK cluster.

1. Hardware fault diagnostics.

2. Driver fault diagnostics.

3. Containerized environment faults diagnostics.

4. cGPU fault diagnostics.

5. Application fault diagnostics.

6. Other diagnostic systems. For example, when a GPU error log system is installed in the cluster, the system can quickly diagnose the cause of faults after receiving application error logs.

The system diagnoses from the lower hardware layer to the upper application layer in sequence because faults in the upper layer may be caused by faults in the lower layer. For example, if an application cannot use GPU resources, an issue may occur in the underlying GPU. This bottom-up diagnostics workflow can avoid interference from faults in lower layers when diagnosing faults in upper layers.

Step 3: Fault isolation

If a fault is detected during the fault diagnostics workflow, fault isolation is performed. The following figure shows the fault isolation workflow.

In the preceding figure, fault isolation consists of scheduling layer isolation and application layer isolation. When an application fault occurs, the application does not affect the use of GPU resources by other applications on the node. Therefore, isolation is not required at the scheduling layer or application layer.

• Scheduling layer isolation:

  • Node isolation: Perform the cordon operation on the node. Driver faults, containerized environment faults, and cGPU faults affect the use of GPUs on the node. You must isolate the node.

  • GPU isolation: Report the IDs of healthy GPUs to the kubelet by using Device Plugin and ignore the IDs of unhealthy GPUs. When a hardware fault occurs, if a single GPU fails without affecting the use of other GPUs on the node, isolating the faulty GPU can reduce GPU resource waste. However, if a single GPU fails and the fault affects the use of other GPUs on the node, node isolation is required.

• Application layer isolation: In addition to scheduling layer isolation, application controllers also need to be isolated in some scenarios.

  • Avoid scheduling application pods that tolerate all taints to the faulty node.

  • Avoid scheduling application pods to the faulty node if nodes or GPUs are already specified for the pods.

  • In elastic training scenarios, the application controller may terminate other healthy pods of a training task on the faulty node and then launch these pods on other nodes. The controller also avoids scheduling other training tasks to the faulty node during the fault and recovery periods.

    For example, during elastic training, if a pod of a training task is restarted due to a GPU-accelerated node fault, the event source of the node reports the fault to the training task controller. The controller identifies the fault and related event in advance and performs a scale-in operation to isolate the faulty node or GPU. During the node fault and recovery periods, the application controller avoids repeatedly migrating training task pods. For more information about how to deploy elastic model training tasks and scale resources for training tasks, see Elastic training based on Horovod.

Step 4: Fault confirmation

After fault isolation is completed, you need to further confirm whether the fault is a false positive.

• If the fault diagnostic result shows a false positive, cancel the recovery workflow.

• If the fault diagnostic result shows a fault, assess whether you can perform operations that may affect your businesses (such as restarting a node) during the recovery workflow based on the business scenario.

Step 5: Fault recovery

After you confirm that the fault must be fixed, initiate the fault recovery workflow. The following figure shows the workflow.

You must select an appropriate recovery method based on the fault.

• Hardware faults, driver faults, and cGPU faults: You must ensure that no applications use the GPU resources on the faulty node. This means you must perform the recovery operation after the node is drained. However, not all types of applications support the node draining operation. If all pods on a node are drained during elastic training, the training task may fail. For example, the actual state of the task may not be saved. Before you drain the node, the system notifies the application controller to migrate the pods that use GPU resources on the node. You are notified after the migration is completed.

• Containerized environment faults: Reinstall NVIDIA Container Toolkit or restart the pod that runs Device Plugin without migrating the applications on the GPU-accelerated node.

• Application faults: Identify and resolve the problem in the application.

Step 6: Fault isolation removal

After the fault is fixed, you must remove fault isolation to ensure that the faulty node or GPU can be used in the production environment as normal and the applications run smoothly.