All Products
Search

This Product

    Container Service for Kubernetes:Enable auto-repair for Lingjun node pools

    Document Center

    Container Service for Kubernetes:Enable auto-repair for Lingjun node pools

    Last Updated:Jun 25, 2026

    After you enable node auto-repair, ACK monitors Lingjun nodes for faults and invokes the Approve O&M operations API to repair them.

    Enabling node auto-repair

    Enable node auto-repair and define rules when creating or configuring a managed node pool. Manage all auto-repair rules on the Self-Healing Rule page. See Auto-repair rule configuration.

    Important

    • Available to allowlisted users only. Contact technical support to enable.

    • Not supported in Apsara Stack regions.

    Method 1: New node pool

    1. On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.

    2. Click Create Lingjun Node Pool. In the Configure Managed Node Pool section, enable the managed feature, then create and enable an auto-repair rule.

      See Create a Lingjun node pool.

    Method 2: Existing node pool

    1. On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.

    2. In the Actions column for the target Lingjun node pool, click image and then select Enable Managed Node Pool (if not yet enabled) or Configure Managed Node Pool (if already enabled). Create an auto-repair rule and enable the feature.

      See Create a Lingjun node pool.

    After enabling the feature, create and manage auto-repair rules on the Inspections and Diagnostics > Self-Healing Rule page.

    1. On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click Inspections and Diagnostics > Self-Healing Rule page.

    2. Click Create Self-Healing Rule, or manage existing rules in the rule list.

    Auto-repair rule configuration

    Important

    • Faults are processed by priority. High-priority faults requiring maintenance are repaired first. Low-priority faults resolvable by reboot are delayed until all high-priority faults are resolved.

    • In the Custom Remediation Flow, if any step has Manual Intervention Required enabled, grant authorization promptly. Otherwise, ACK pauses the current and subsequent steps, and the node may remain impaired.

    • After node auto-repair is enabled, ACK monitors abnormal events and invokes the Approve O&M operations capability to repair underlying faults on Lingjun nodes. Monitor Lingjun node health and cluster status closely to prevent prolonged service disruptions.

    1. Access the rule configuration page

    1. On the node pool creation or configuration page, or on the Self-Healing Rule page, click Create Self-Healing Rule.

    2. Specify a rule name and configure the rule:

    2. Configure sub-rules

    a. Select Fault Type

    Node auto-repair monitors CloudMonitor events from Lingjun nodes and repairs faults via the Lingjun Approve O&M operations API.

    Fault types and remediation actions:

    Fault name

    Lingjun event name

    Kubernetes condition

    Fault description

    Remediation action

    Lingjun node hardware failure (Repair required)

    Node.HardwareFault.HOST:Inquiring

    resourceType: Node

    • Type: Node.HardwareFault.HOST

    • Status: True

    • Reason:Node.HardwareFault.HOST:Inquiring

    • Message: <Fault details>

    The node requires maintenance for a hardware failure. Local data may be lost after maintenance. If container runtime and kubelet directories are on local disks, remove and re-add the node after maintenance.

    Repair the node.

    Important

    Local disk data may be lost during repair. Back up data in advance.

    Priority: High

    Lingjun node underlying error (Restart required)

    Node.FaultNeedReboot.HOST:Inquiring

    resourceType: Node

    • Type: Node.FaultNeedReboot.HOST

    • Status: True

    • Reason:Node.FaultNeedReboot.HOST:Inquiring

    • Message: <Fault details>

    A failure occurred on the node. Restart it promptly to attempt remediation.

    Restart the node.

    Priority: Medium

    b. Custom Remediation Flow

    All steps support manual intervention via Manual Intervention Required (options: Disable or Authorize by Node Label). When set to Authorize by Node Label, ACK waits for authorization before executing the step.

    Step

    Description

    Isolate Node

    Marks the faulty node as unschedulable (spec.unschedulable = true) to prevent scheduling new workloads to it.

    Node draining
    Requires Isolate Node to be enabled first.

    ACK evicts workloads from the failed node one at a time. Default drain timeout: 300 seconds (adjustable).

    • Node draining adheres to the PDB protocol. These pods are not evicted:

      • Pods managed by a DaemonSet.

      • Pods managed by an OpenKruise Advanced DaemonSet.

      • Pods with the alibabacloud.com/evict-pod=true label.

    • If workloads cannot be evicted within the timeout, ACK skips eviction and continues.

    Auto repair

    When enabled, ACK performs the repair action for the fault type. See Fault types and repair behaviors.

    Uncordon Node
    Requires Isolate Node to be enabled first.

    Restores the node to schedulable (spec.unschedulable = false).

    If the node was unschedulable before the repair process, this step does not restore it.

    Node auto-repair events

    When auto-repair is enabled, ACK converts Lingjun node fault events into Kubernetes Events and Conditions , and generates Events to track repair progress.

    Common auto-repair events

    Event (reason)

    Level

    Description

    DetectedLingJungCMSEvent

    Warning

    A Lingjun node fault was detected or an existing fault status was updated.

    RepairPlanCreated

    Warning

    An auto-repair plan was created.

    WaitingApproveProcedure

    Warning

    The current step is paused, awaiting manual authorization. The event details the required action and how to grant it.

    RepairPlanProcedureWaitingForApproval

    Warning

    The authorization label was added to the node.

    RepairPlanStatusUpdated

    Normal

    The auto-repair plan status was updated. Records the current step and its result.

    ApproveLingjunOperation

    Normal

    The Lingjun control plane was authorized to perform the repair.

    NodeNotRecovery

    Warning

    Triggered if a node fails to recover after repair. ACK checks node status every 30 minutes post-repair.

    RepairPlanCompleted

    Warning

    The auto-repair plan finished. The event indicates whether the plan succeeded or failed.

    Events for special scenarios

    These events occur in specific scenarios.

    Event (reason)

    Level

    Description

    NodeRecoveryNeedSilence

    Normal

    The latest auto-repair plan on this node succeeded, but the current fault type differs from the previous one. The system waits 10 minutes before creating a new plan.

    NodeRecoveryHasProcessed

    Warning

    The latest auto-repair plan processed this fault successfully, but the node has not recovered. Investigate manually.

    RepairPlanFailed

    Warning

    The auto-repair plan for the current fault failed and the node has not recovered. Investigate manually.

    RepairPlanStatusUpdateFailed

    Warning

    Failed to update the auto-repair plan status.

    AddInquiringLabelFailed

    Normal

    Failed to set the authorization label. The auto-repair process was aborted.

    Configure alerts

    After you enable node auto-repair, you must activate the Alert Rules for Node Pool O&M Events and the Cluster Node Auto Repair Alert Rule Set to receive alerts when anomalies occur.

    See Alarm management for Container Service.