Detect and Auto-Recover from GPU-HPN Node Faults - Container Compute Service

During AI training, a GPU node fault can cause tasks to hang or terminate abnormally. Alibaba Cloud Container Service for Kubernetes (ACS) monitors GPU-HPN nodes and, when a node fails, marks it with a taint and reports the fault through Kubernetes conditions and pod events. This topic describes the fault types, how to read fault signals, and the node recovery process.

Node fault handling process

ACS continuously runs health checks on GPU-HPN nodes. When a node fails, follow this process to repair it using the ACS self-healing mechanism.

Fault notification ACS reports the fault cause through events and node conditions, and adds a taint to prevent new pods from being scheduled to the node.
Drain the faulty node After receiving the fault notification, evict all pods from the faulty node. Use acs-instance-helper to automate pod eviction. For more information, see Configure automatic rotation for instances with hardware exceptions.
Repair the faulty node After all pods are evicted, ACS automatically starts the repair process.
Node self-recovery After ACS repairs the node, the related taints and conditions are automatically restored to their normal state, and new pods can be scheduled to the node.

Fault types

The following table describes the fault types reported in node conditions and pod events.

Fault type	Description
`NodeBroken`	The entire GPU-HPN node has failed.
`GPUCardBroken`	A GPU card-level failure has occurred.
`NodeMaintenance`	The GPU-HPN node is undergoing a system upgrade or O&M.

Fault notifications

Important

To ensure prompt fault handling, configure alert conditions using kube-eventer. For more information, see kube-eventer.

When a GPU-HPN node is faulty, ACS provides fault information through conditions on the Node object and events on the affected pods.

Node conditions

ACS uses two condition types on the Node object: NodeAnomaly and FaultHealing.

NodeAnomaly condition

When a fault is detected, ACS:

Adds a label to the node for filtering:

metadata:
  labels:
    alibabacloud.com/node-anomaly: "true"

Adds a NoSchedule taint to prevent new pods from being scheduled to the node:

spec:
  taints:
  - effect: NoSchedule
    key: alibabacloud.com/node-anomaly
    timeAdded: "2024-10-16T06:09:27Z"

Records detailed fault information in the NodeAnomaly condition:

status:
  conditions:
  - lastHeartbeatTime: "2024-10-16T06:09:31Z"
    lastTransitionTime: "2024-10-16T06:09:31Z"
    message: The node has encountered an anomaly.
    reason: NodeBroken
    status: "True"
    type: NodeAnomaly

To query the NodeAnomaly condition for all nodes:

kubectl get nodes -o custom-columns='NAME:.metadata.name,ANOMALY:.status.conditions[?(@.type=="NodeAnomaly")].status,REASON:.status.conditions[?(@.type=="NodeAnomaly")].reason'

To list all nodes with the anomaly label:

kubectl get nodes -l alibabacloud.com/node-anomaly=true

The following table describes the NodeAnomaly condition fields.

Field	Description	When it is updated
`type`	`NodeAnomaly`. Indicates a node anomaly.	Static field. Does not change during the node lifecycle.
`status`	Whether a fault exists. `True`: a fault exists. `False`: no fault exists.	Updated when the node fault status changes.
`reason`	Fault type. See Fault types for the possible values.	Updated when the node fault status changes.
`message`	Detailed fault information.	Updated when the node fault status changes.
`lastTransitionTime`	The time when the fault status last changed.	Updated when the node fault status changes.
`lastHeartbeatTime`	The regularly updated heartbeat time.	Updated when the fault status changes, or if more than five minutes have passed since the last update.

FaultHealing condition

ACS records self-healing progress in the FaultHealing condition. This condition is updated as the repair process progresses.

status:
  conditions:
  - lastHeartbeatTime: "2025-03-24T11:14:48Z"
    lastTransitionTime: "2025-03-24T11:14:48Z"
    message: node fault healing success
    reason: Success
    status: "False"
    type: FaultHealing

To monitor self-healing progress:

kubectl get nodes -o custom-columns='NAME:.metadata.name,HEALING:.status.conditions[?(@.type=="FaultHealing")].status,REASON:.status.conditions[?(@.type=="FaultHealing")].reason,MESSAGE:.status.conditions[?(@.type=="FaultHealing")].message'

The following table describes the FaultHealing condition fields.

Field	Description
`type`	`FaultHealing`. Indicates that the node is in the self-healing process.
`status`	Whether self-healing is in progress. `True`: self-healing is in progress. `False`: self-healing is complete.
`reason`	Self-healing status. `Success` or `Finished`: complete. `InProgress`: in progress. `Failed`: self-healing failed.
`message`	Detailed self-healing progress information.
`lastTransitionTime`	Updated when the self-healing progress changes.
`lastHeartbeatTime`	Updated when the self-healing progress changes.

Pod events

Important

After receiving the fault notification, evict pods from the faulty node as soon as possible. ACS automatically starts the repair process after all pods are evicted. Use acs-instance-helper to automate pod eviction. For more information, see Configure automatic rotation for instances with hardware exceptions.

ACS generates a Warning event for each pod running on the faulty node.

reason: NodeBroken
type: Warning
message: 'The pod is proposed to be evicted at 2024-10-16 07:21:54 +0000 UTC, reason: xxx'

To view fault-related events on a specific node:

kubectl get events --field-selector involvedObject.name=<node-name> | grep -i "NodeBroken\|GPUCardBroken\|NodeMaintenance"

The following table describes the pod event fields.

Field	Description
`type`	Static field. The value is `Warning`.
`reason`	Fault type. See Fault types for the possible values.
`message`	Detailed information about the proposed eviction.