Common maintenance and system events for local disk instances - Elastic Compute Service

Identify and respond to maintenance scenarios and system events for ECS instances with local disks.

View and monitor system events

View system events:

View events in the ECS console or by using Alibaba Cloud CLI. See Query and handle events.
View events in the CloudMonitor console. See View system events.

Monitor system events:

Set up event notifications to monitor underlying environment changes and automate maintenance responses.

Configure alert rules in CloudMonitor. See Subscribe to system event notifications.
Use a DingTalk chatbot. See Send event notifications with a DingTalk chatbot.
For ECS Bare Metal instances, install the xdragon_hardware_detect_plugin plug-in to periodically check local disk health. See Install the monitoring plug-in.

Common maintenance scenarios and related system events

The following figure shows common maintenance scenarios and related system events for local disk instances.

Note

Customize the automatic recovery method by modifying instance maintenance attributes. For example, if set to automatic redeployment, the instance enters a redeployment scenario by default during automatic recovery. See Modify instance maintenance attributes.

See the following sections for details on each scenario:

Scenario ①: SystemMaintenance.Reboot
Scenario ②: SystemMaintenance.Redeploy
Scenario ③: SystemFailure.Reboot
Scenario ④: SystemFailure.Redeploy
Scenario ⑤: Disk:ErrorDetected, SystemMaintenance.IsolateErrorDisk, SystemMaintenance.ReInitErrorDisk, SystemMaintenance.RebootAndIsolateErrorDisk, or SystemMaintenance.RebootAndReInitErrorDisk
Scenario ⑥: SystemMaintenance.StopAndRepair

Note

Before maintenance, ensure service availability and back up your data. For example, switch traffic at the application layer, remove the ECS instance from a Server Load Balancer (SLB) instance, and back up disk data.

Scenario ①

Handle a SystemMaintenance.Reboot event:

Receive a notification that your instance is scheduled for restart.
Respond to the event as needed.
- To change the scheduled time window, see Modify scheduled restart time.
- Restart the instance within the user operation window. See Restart an instance.
  
  Note
  Restart the instance from the ECS console or by calling the RebootInstance API. Restarting from within the operating system has no effect.
- Wait for the system to automatically restart the instance.
Verify that the instance and your applications work as expected.

For SystemMaintenance.Reboot event states, see System event summary. For state transition diagrams, see States and windows of system events.

Scenario ②

Handle a SystemMaintenance.Redeploy event:

Receive a notification that your local disk instance is scheduled for redeployment.
Complete the prerequisite tasks, such as modifying the /etc/fstab file and backing up data.

See the Prerequisites section of Redeploy instance.
Respond to the event.

After backing up data, respond to the event to trigger instance migration and local disk replacement. See Redeploy instance.

Note
Redeploying a local disk instance migrates it to a new host and re-initializes the local disk. All data on the local disk is erased.
Verify that the instance and your applications work as expected, and synchronize data as needed.

For SystemMaintenance.Redeploy event states, see System event summary. For state transition diagrams, see States and windows of system events.

Scenario ③

Handle a SystemFailure.Reboot event:

The system automatically restarts the instance.
You receive a notification that the instance is restarting.

The instance is restarting automatically. No action is required.
Verify that the instance and your applications work as expected.

For SystemFailure.Reboot event states, see System event summary. For state transition diagrams, see States and windows of system events.

Scenario ④

Handle a SystemFailure.Redeploy event:

Receive a notification that your local disk instance is scheduled for redeployment.
Complete the prerequisite tasks, such as modifying the /etc/fstab file and backing up data.

See the Prerequisites section of Redeploy instance.
Respond to the event as needed.

After backing up data, respond to the event to trigger instance migration and local disk replacement. See Redeploy instance.

Note
Redeploying a local disk instance migrates it to a new host and re-initializes the local disk. All data on the local disk is erased.
Verify that the instance and your applications work as expected, and synchronize data as needed.

For SystemFailure.Redeploy event states, see System event summary. For state transition diagrams, see States and windows of system events.

Scenario ⑤

In Scenario ⑤, you can redeploy the instance to another host or replace only the damaged disk. When replacing a damaged disk:

Not all disks can be isolated. Isolation is only possible when the system event operation includes disk isolation.
Disk isolation and repair are independent operations. Isolation is a prerequisite for repair, but does not guarantee repair is possible. Not all instances support local disk repair. You can initiate a repair only after receiving a disk recovery notification from Alibaba Cloud.

Redeployment quickly restores local disk availability but erases all data on local disks. See Redeploy instance.
Replacing a damaged disk preserves data on other local disks. The workflow:
1. Receive notifications about a disk failure and scheduled isolation.
2. Complete the prerequisite tasks, such as modifying the /etc/fstab file and backing up data.
3. If the system event includes a disk isolation event, respond to authorize the isolation.
4. If the system event includes a Reboot event, restart the instance.
5. Alibaba Cloud removes the damaged disk, inserts a new one, and sends a disk recovery notification.
6. If the system event includes a disk recovery event, respond to authorize the recovery.
7. If the system event includes a Reboot event, restart the instance.
Note
Replacing a damaged disk requires cooperation between you and Alibaba Cloud. See Isolate/repair local disks and Isolate damaged local disks (CLI).

The following figure shows supported event states and transitions for damaged disk replacement.

Scenario ⑥

In Scenario ⑥, you can redeploy the instance to another host or perform an in-place repair. When performing an in-place repair:

An in-place repair cannot guarantee 100% data integrity or success rate. Back up critical data before authorizing the repair.
Stop-and-repair is not supported on all instances with local disks.
During repair, the instance cannot start, but billing continues.
The in-place repair cycle is 14 business days. You can terminate the repair by redeploying or releasing the instance.

Redeployment quickly restores local disk availability but erases all data on local disks. See Redeploy instance.
The in-place repair workflow:
1. Receive a system event notification about an in-place repair for your local disk instance.
2. Respond to the event as needed.
  - Within the user operation window, stop the instance and authorize the repair.
  - Wait for the system to automatically stop the instance and perform the repair.
3. Alibaba Cloud repairs the host hardware and sends a repair completion event.
4. Verify that the instance and your applications work as expected, and synchronize data as needed.

For SystemMaintenance.StopAndRepair event states, see System event summary. For state transition diagrams, see States and windows of system events.

References

Call the AcceptInquiredSystemEvent API to accept the default operation for a system event and authorize execution.