Configure automatic instance rotation for hardware failures - Container Compute Service

How it works

When an ACS instance experiences scheduled node maintenance or a hardware fault such as a damaged GPU, service stability and performance can be affected. The acs-instance-helper component automates fault handling:

Automatic fault monitoring: The component continuously monitors the fault Condition of Pods. The underlying infrastructure automatically reports this signal for events like GPU failures, host machine faults, or scheduled node maintenance and reboots.
Maintenance window alignment: The component determines when to act based on the fault handling deadline reported by the underlying node and an optional maintenance window. If the deadline allows, the component waits for the predefined maintenance window before proceeding.
Triggered rotation update: For stateless applications such as Deployments and CloneSets, the component uses an online scale-up strategy (scale up first, then destroy) to rotate Pods on the faulty node.

Important
For non-online applications, acs-instance-helper directly evicts the Pods on the instance after detecting the fault condition.

Prerequisites

Your ACS cluster is version 1.28 or later.
ACK Virtual Node is installed, and is v2.16.0 or later. For more information, see ACK Virtual Node.

Install the component

In the ACS console, click the name of your target cluster. In the navigation pane on the left, choose Applications > Helm.
On the Helm page, click Create.
1. Basic Information: In the Chart search box, enter acs-instance-helper and select it from the results.
2. Parameters: For Chart Version, select the latest version.

Configure global settings for acs-instance-helper (Optional)

You can also configure a maintenance window and add support for custom workload types.

Console

In the navigation pane on the left, choose Configurations > ConfigMaps.
On the ConfigMaps page, click Create from YAML. Copy the following manifest into the Template area and click Create.

kubectl

Get the cluster KubeConfig and connect to the cluster by using kubectl.
Save the following YAML content as the acs-instance-helper-global-configmap.yaml file, and then run the kubectl apply -f acs-instance-helper-global-configmap.yaml command.

apiVersion: v1
kind: ConfigMap
metadata:
  name: acs-instance-helper-global-config
  namespace: kube-system
data:
  customOnlineWorkloads: foo.io/SomeWorkload,bar.io/AnotherWorkload
  hardwareFaultEvictionSeconds: "60"
  maintenanceTime: "2025-10-09T10:00:00+08:00"
  maintenanceDuration: "4h"
  maintenanceWeeklyPeriod: "Saturday,Sunday"
  # maintenanceRecurrence: "FREQ=WEEKLY;BYDAY=SA,SU"  # Maintenance window: every Saturday and Sunday

Expand the following section for parameter descriptions.

Parameters

Key	Description	Example
`customOnlineWorkloads`	Marks a workload as an "online service" type that uses the online scale-up strategy (scale up first, then destroy) to rotate Pods on faulty nodes. By default, the component supports the Deployment workload type. You can use this parameter to add support for custom workload types. Important Ensure that your custom workload controller can maintain the specified number of replicas (for example, by automatically creating new ones if the count is too low). Seamless rotation is not guaranteed for all custom workloads. When you enable this feature for a custom workload, test it thoroughly.	`foo.io/SomeWorkload,bar.io/AnotherWorkload`
`hardwareFaultEvictionSeconds`	For services that use the "scale up and evict" strategy, this parameter defines the waiting period (in seconds) between scale-up completion and Pod eviction on the faulty instance. The default value is `"300"` (5 minutes). The value must be a string, for example, `"60"`.	`"60"`
`maintenanceTime`	Enables the maintenance window for the cluster and sets its start time in `RFC3339` format. Use an explicit timezone identifier, such as `+08:00` or `UTC`.	`2025-10-09T10:00:00+08:00` Declares that the maintenance window starts at 10:00 AM on October 9, 2025 (UTC+8).
`maintenanceDuration`	Specifies the duration of each maintenance window. This parameter takes effect only when `maintenanceTime` is configured. The value must be a string. Formats such as `"3"`, `"3h"`, and `"3H"` are supported. The default value is `"3"`, which means 3 hours.	`"4h"`
`maintenanceWeeklyPeriod`	Specifies the days of the week for maintenance. This parameter takes effect only when `maintenanceTime` is configured. Valid values are `Monday`, `Tuesday`, `Wednesday`, `Thursday`, `Friday`, `Saturday`, and `Sunday`. Separate multiple values with commas. If this parameter is configured, it overrides the value of `maintenanceRecurrence`.	`Saturday,Sunday`
`maintenanceRecurrence`	Defines a custom maintenance schedule using `RFC5545` recurrence rule syntax. This parameter takes effect only when `maintenanceTime` is configured. Currently, only `FREQ=WEEKLY` is supported. The `COUNT` and `UNTIL` parameters are not supported. If `maintenanceWeeklyPeriod` is also configured, this parameter is ignored.	`FREQ=WEEKLY;BYDAY=SA,SU`

Fault resolution timing depends on both the fault handling deadline and the configured maintenance window. If a maintenance window is available before the deadline, acs-instance-helper prioritizes repairs during that window. If the process is not completed within one window, remaining operations continue in the next available window.

Create and configure a workload

Enable the fault handling feature for your workload by configuring an annotation.

The fault handling feature performs rotation by repeatedly attempting evictions by using the Eviction API rather than by directly deleting Pods on faulty instances. You can configure a PodDisruptionBudget (PDB) policy to control the concurrency of evictions and prevent service disruptions. For more information, see Use a PDB to control Pod eviction concurrency.

Console

In the navigation pane on the left of your target cluster, choose Workloads > Deployments.

On the Deployments page, click Create from YAML. Copy the following content into the Template area and click Create.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hardware-fault-helper-example
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hardware-fault-helper-example
  template:
    metadata:
      labels:
        app: hardware-fault-helper-example
      annotations:
        # Key annotation: Enables the fault handling feature for the workload.
        "ops.alibabacloud.com/enable-hardware-fault-helper": "true"
    spec:
      containers:
        - image: registry-cn-hangzhou.ack.aliyuncs.com/dev/hello-world:v1
          name: main-container
          resources:
            limits:
              cpu: 100m
              memory: 100Mi
      restartPolicy: Always

In the dialog box that appears, find the target stateless application and click View . Confirm that the Pod status is Running.

kubectl

Save the following YAML content as app.yaml and run the command kubectl apply -f app.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hardware-fault-helper-example
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hardware-fault-helper-example
  template:
    metadata:
      labels:
        app: hardware-fault-helper-example
      annotations:
        # Key annotation: Enables the fault handling feature for the workload.
        "ops.alibabacloud.com/enable-hardware-fault-helper": "true"
    spec:
      containers:
        - image: registry-cn-hangzhou.ack.aliyuncs.com/dev/hello-world:v1
          name: main-container
          resources:
            limits:
              cpu: 100m
              memory: 100Mi
      restartPolicy: Always

Confirm that the Pod status of the target application is Running.
```
kubectl get pods -l app=hardware-fault-helper-example
```

Simulate a fault scenario

In production, the Condition is added automatically by the underlying control plane. In this section, you manually inject a Condition into a Pod to simulate a fault scenario.

Simulate the fault: Replace POD_NAME with the actual name of your Pod to inject a hardware fault Condition.

The fault handling deadline is specified in the message field.

kubectl patch pod POD_NAME --type='merge' --subresource=status -p='{
  "status": {
    "conditions": [
      {
        "type": "Interruption.HardwareFault",
        "status": "True",
        "reason": "MockForTest",
        "message": "Underlying infrastructure issue [Reboot] scheduled at 2099-03-12T09:00:00.000+08:00",
        "lastProbeTime": "'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'",
        "lastTransitionTime": "'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'"
      }
    ]
  }
}'

Observe the scale-up: After you inject the fault, acs-instance-helper triggers a scale-up according to the maintenance window configuration. If no window is configured, it triggers the scale-up immediately. A new Pod is created, and the original workload's status remains unaffected.

kubectl get pods -l app=hardware-fault-helper-example

Expected output:

NAME                                             READY   STATUS    RESTARTS   AGE
hardware-fault-helper-example-7cf4cf96c5-xxxxx   1/1     Running   0          2m21s
hardware-fault-helper-example-7cf4cf96c5-yyyyy   1/1     Running   0          36s # The newly scaled-up Pod

Check the scale-up event: Check the events for the faulty Pod. You should see a NewInstanceCreationTriggered event, which confirms that the scale-up was triggered by hardware-fault-helper.

kubectl describe po POD_NAME

Expected output:

...
  Normal  NewInstanceCreationTriggered  62s    hardware-fault-helper  controller default/hardware-fault-helper-example-7cf4cf96c5 (apiVersion:apps/v1, kind:ReplicaSet) will create a new instance

Check the eviction event: After the hardwareFaultEvictionSeconds waiting period, the faulty Pod is taken offline. It enters the Terminating state before being deleted. You can also observe an event for this action.

kubectl describe po POD_NAME

Expected output:

...
  Warning  InstanceEvictedGracefully     2s     hardware-fault-helper  pod is deleted due to hardware fault
  Normal   Killing                       1s     kubelet                Stopping container main-container

Confirm recovery: Finally, the faulty Pod is completely replaced, leaving only the newly created Pod.

kubectl get pods -l app=hardware-fault-helper-example

Expected output:

NAME                                             READY   STATUS      RESTARTS   AGE
hardware-fault-helper-example-7cf4cf96c5-yyyyy   1/1     Running     0          5m5s

Billing

Installing the acs-instance-helper component deploys a Deployment with two replicas in your cluster. Each replica consumes 1 vCPU and 2 GiB of memory from your cluster, which will incur fees. For more information about billing, see ACS computing power billing.

FAQ

Control Pod eviction with a PDB

To maintain high availability during Pod evictions for node draining and autoscaling, configure a PodDisruptionBudget (PDB) policy. The PDB controls eviction concurrency with the following parameters:

maxUnavailable: The maximum number of Pods that can be unavailable during the eviction process.
minAvailable: The minimum number of Pods that must remain available during the eviction process.

The following example ensures that at least one Pod remains available during an eviction:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
  namespace: YOUR_NAMESPACE # Specify the namespace where the policy applies. Defaults to `default` if not specified.
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: app