All Products
Search
Document Center

Elastic Compute Service:Overview

Last Updated:Mar 22, 2024

System events are defined by Alibaba Cloud to record and send you resource information notifications, such as the execution states of O&M tasks, resource exceptions, and changes in the status of resources.

Note

Many Alibaba Cloud services, such as Elastic Compute Service (ECS), ApsaraDB RDS, and Server Load Balancer (SLB), support system events. This topic describes only ECS system events. For information about the system events of other Alibaba Cloud services, see the relevant documentation.

Limits

Retired instance families do not support system events. For more information, see Retired instance types.

Usage scenarios of system events

  • Notification of risks and exceptions

    After system events that are not displayed in the ECS console are triggered, Alibaba Cloud pushes the events to the ECS console. The events include the events that affect the availability and performance of ECS resources, such as SystemMaintenance.Reboot events among scheduled O&M events and InstanceExpiration.Stop events among instance billing events. For specific critical system events, Alibaba Cloud sends additional emails or internal messages. You can handle the events in the ECS console or by calling API operations. We recommend that you handle the system events at the earliest opportunity to ensure resource availability and performance. For more information, see Query and handle ECS system events.

    For example, when a subscription instance is about to expire, the ECS console prompts you to renew the instance within a specific period of time to ensure service continuity.

  • Automated O&M

    The states of system events that are displayed in the ECS console are defined to help you understand the execution states of system O&M tasks. Meanwhile, new system events and changes in the status of system events are reported to CloudMonitor. This way, you can build an event-driven automated O&M system based on your business requirements. For more information about the states of events, see the States and windows of system events section in this topic.

    Note

    Each event state corresponds to a CloudMonitor event. For example, the Executing and Executed states that the InstanceFailure.Reboot ECS event code supports correspond to the Instance:InstanceFailure.Reboot:Executing and Instance:InstanceFailure.Reboot:Executed CloudMonitor events.

    Some status change events are not displayed in the ECS console and cannot be handled in the ECS console or by calling API operations. Examples of the events include events that indicate changes in the status of instances and interruptions of preemptible instances. The states of the system events are not defined in ECS. However, the events are still reported to CloudMonitor when the events are triggered. This way, you can build an event-triggered automated O&M system based on your business requirements.

    For example, status change events are triggered when you start or stop instances. The events do not indicate risks or exceptions. If you want to log your operations to your system, you can configure event notifications for status change events and use the alert callback feature to write the startup and stop information of instances to operation logs.

System event categories

System events are defined by Alibaba Cloud to record and send you resource information notifications. System events are classified into the categories that are described in the following table based on the causes of events.

Note

For information about the system event categories supported by ECS and how to handle ECS system events, see Summary.

Category

Description

Displayed in the ECS console

Scheduled O&M events

Alibaba Cloud may need to upgrade host software for security reasons or to predict and handle failure risks that lie in underlying host hardware and software. In these cases, if O&M tasks to be executed by Alibaba Cloud may affect the availability or performance of your ECS resources, Alibaba Cloud triggers and sends scheduled O&M events in advance to notify you of task details, such as execution times, objects, and impacts. After you receive a scheduled O&M event, you can handle the O&M event during an off-peak period within the event execution window to minimize business impacts.

Note

Scheduled O&M events, also known as proactive O&M events, are based on the O&M experience of Alibaba Cloud on millions of servers, the ability to serve tens of thousands of large enterprise customers, and the cutting-edge machine learning algorithms of Alibaba DAMO Academy to predict and handle failure risks that lie in underlying host hardware or software. When failure risks on the host cannot be prevented, Alibaba Cloud notifies you in advance by using scheduled O&M events. This way, you can switch your business before failures occur. If you do not respond to the scheduled O&M events at the earliest opportunity, your ECS instances may break down or restart when failures occur.

Yes

Note

When scheduled O&M events are triggered for instances of big data instance families (excluding the d3c instance family) or instance families that are equipped with local SSDs (excluding the i4p instance family), the events are displayed on the Local Disk-based Instance Events page. For information about local disk-based instance events, see O&M scenarios and system events for instances equipped with local disks.

Unexpected O&M events

This category of system events is triggered when ECS instances restart or break down due to unexpected issues, such as kernel panic, out-of-memory (OOM) errors, or hardware or software failures in underlying hosts. Alibaba Cloud sends the events after they are triggered and restores affected ECS resources at the earliest opportunity. Alibaba Cloud also notifies you of the execution states of system O&M tasks related to the events.

Note

In most cases, unexpected O&M events refer to sudden downtime or restarts of ECS instances due to unpredictable failures of the underlying hosts or kernel errors in the operating systems of the ECS instances.

  • ECS instance downtime or restart events caused by host failures (SystemFailure.Reboot) are occasional and inevitable. If the Service Level Agreement (SLA) for a single instance is violated, Alibaba Cloud will pay compensation based on the SLA of the related service.

  • In most cases, ECS instance restart events caused by operating system kernel errors (InstanceFailure.Reboot) are caused by applications. You can capture dump files to analyze the causes.

Yes

Note

When unexpected O&M events are triggered for instances of big data instance families (excluding the d3c instance family) or instance families that are equipped with local SSDs (excluding the i4p instance family), the events are displayed on the Local Disk-based Instance Events page. For information about local disk-based instance events, see O&M scenarios and system events for instances equipped with local disks.

Local disk-based instance events

This category of system events includes system events that are triggered for local disks and instances equipped with local disks. The system events that are triggered for local disks include system events triggered when local disks are damaged. The system events that are triggered for instances equipped with local disks include system events triggered when instances equipped with local disks fail due to local disk damages or when the hardware or software of underlying hosts fails for instances equipped with local disks.

Note

Local Disk-based Instance Events are not a system event category and are used only to display scheduled or unexpected O&M events for instances of big data instance families (excluding the d3c instance family) or instance families that are equipped with local SSDs (excluding the i4p instance family) and make the events easy to handle. For more information about local disk-based instance events, see O&M scenarios and system events for instances equipped with local disks.

Yes

Performance limited events of burstable instances

This category of system events is triggered when burstable instances exhaust their CPU credits and start to run at or near the baseline CPU utilization. The system events may affect instance management, instance O&M, and the operation of applications and result in issues such as slow access and delays.

Yes

Instance security events

This category of system events is triggered when instances face security threats. For example, instance security events are triggered when instances are under DDoS attacks or when blackhole filtering is triggered for instances.

Yes

Instance migration events due to upgrades at the underlying layer

This category of system events is triggered when instances need to be migrated from specific regions and zones due to an infrastructure upgrade plan of Alibaba Cloud. You can migrate instances based on the system events.

Yes

Status change events

This category of system events is triggered when operations, such as Start and Stop, on instances cause changes in the status of the instance lifecycle or when instance attribute changes cause changes in the status of the instance lifecycle or other status changes. Status change events are classified into the following categories:

  • Lifecycle status change events: For example, lifecycle status change events are triggered when instances enter a different state, when preemptible instances are interrupted, and when snapshots are created.

  • Other attribute change events: For example, other attribute change events are triggered when the performance mode of burstable instances is changed or when subscription disks are changed into pay-as-you-go disks.

  • Lifecycle status change events are not displayed in the ECS console.

  • Specific other attribute change events are displayed in the ECS console.

System event severities

The following severities are assigned to system events based on the impacts of the system events on the normal operation of instances:

  • Critical: Critical system events may result in instance unavailability and must be handled at the earliest opportunity. For example, a critical system event is triggered when resources are released due to an overdue payment or when an instance is redeployed due to an instance error.

  • Warning: Warning system events affect your business. For example, a warning system event is triggered when a burstable instance cannot burst above its performance baseline. You must take note of the events or handle the events when appropriate.

  • Notification: Notification system events do not affect your business. For example, a notification system event is triggered when a snapshot is created for a disk. You can choose whether to pay attention to notification system events.

States and windows of system events

The following table describes the states defined for system events that are displayed in the ECS console.

Note

For information about the supported states of different system events, see the "CloudMonitor event" columns of tables in Summary.

Event state

Attribute

Description

Inquiring

Intermediate

The O&M task related to the system event is pending authorization. After you authorize the task to be executed, the event enters the Executing state.

Scheduled

Intermediate

The O&M task related to the system event is scheduled and pending execution. When the O&M task is executed, the event enters the Executing state.

Executing

Intermediate

The O&M task related to the system event is being executed.

Executed

Stable

The O&M task related to the system event is complete.

Avoided

Stable

The impacts of the system event are prevented because the affected instance is migrated within the user operation window.

Failed

Stable

The O&M task related to the system event failed.

Canceled

Stable

The O&M task related to the system event is automatically canceled.

The following figure shows the typical transitions between event states.

image

System events have the following windows:

  • User operation window

    The user operation window of a system event starts when the event is sent and ends at the time when the related O&M task is executed as scheduled. You can manually execute the O&M task within the user operation window or wait for the system to automatically execute the task. Take note of the following items about the lengths of user operation windows:

    • In most cases, the user operation window of a scheduled O&M event ranges from 24 to 48 hours.

      Note

      The lengths of user operation windows are unlimited for system events in the Inquiring state. The O&M tasks related to the events can start only after you authorize the tasks to be executed.

    • In most cases, unexpected O&M system events caused by failures or unauthorized operations do not have a user operation window.

    • For system events indicating that subscription instances are about to expire, the window is 3 days.

    • For system events indicating that pay-as-you-go instances are about to be stopped due to overdue payments, the window is less than 1 hour.

  • Event execution window

    The execution window of a system event starts when the related O&M task is executed and ends when the task is complete. Take note of the following items about the lengths of event execution windows:

    • For system events such as failure recovery events, the window is within 10 minutes.

    • Unexpected O&M events caused by failures or unauthorized operations have a short event execution window.

Formats of ECS event codes and CloudMonitor event names

ECS event codes and CloudMonitor events follow specific naming conventions for easy understanding.

  • ECS event codes indicate the event causes and impacts on resources and are in the <Event cause>.<Event impact> format.

  • CloudMonitor event names indicate resource types, event causes, event impacts on resources, and event states and are in the <Resource type>:<Event cause>.<Event impact>:<Event state> format.

Note

ECS event codes and CloudMonitor event names may include only some of the preceding information. For example, a CloudMonitor event name of Disk:ErrorDetected:Executing indicates that a disk is damaged and excludes information about the impacts on resources.

The following table describes some examples of ECS event codes and CloudMonitor event names.

Note

An ECS event code of Undefined indicates that ECS events are not displayed in the ECS console and cannot be handled in the ECS console or by calling API operations. For more information about ECS system events, see Summary.

Category

Example ECS event code

Example CloudMonitor event name

Description

Scheduled O&M events

SystemMaintenance.Reboot

Instance:SystemMaintenance.Reboot:Inquiring

  • Resource type: Instance indicates ECS instance.

  • Event cause: SystemMaintenance indicates that Alibaba Cloud proactively initiates a system O&M task.

  • Event impact: Reboot indicates that the instance is restarted while the O&M task is being executed.

  • Event state: Inquiring indicates that the O&M task related to the event is pending authorization and the instance can be restarted only after you authorize the task to be executed.

Unexpected O&M events

ErrorDetected

Disk:ErrorDetected:Executing

  • Resource type: Disk indicates cloud disk.

  • Event cause: ErrorDetected indicates that the local disk is damaged.

  • Event state: Executing indicates that the damaged local disk has not been repaired.

Lifecycle state change events

Snapshot:CreateSnapshotCompleted

Snapshot:CreateSnapshotCompleted

  • Resource type: Snapshot indicates snapshot.

  • Event cause: CreateSnapshotCompleted indicates that the snapshot is created.

Operations that can be performed on system events

Operation

Description and references

Understand system events

To learn about system events and understand the event codes, names, severities, usage scenarios, limits, states, and name formats, see this topic.

View system events

You can view system events in the ECS or CloudMonitor console or by using Alibaba Cloud CLI.

Handle system events

For specific critical system events, such as system events that affect the availability and performance of ECS resources, we recommend that you handle the events as suggested in the ECS console or by calling API operations at the earliest opportunity to ensure service availability.

Monitor system events

To ensure the stability of services that run on ECS instances and automate O&M, we recommend that you configure event notifications to be notified of underlying environment changes. After you configure event notifications, the system uses the notification methods that you specify to send you notifications.

Modify system event-related settings

You can modify system event-related settings based on your business requirements.

  • You can modify the maintenance attributes of an instance to configure whether to restart or redeploy the instance after a system event is handled. For more information, see Modify instance maintenance attributes.

  • For scheduled system events that require instances to be restarted, you can specify the restart time of the instances after the scheduled system events are handled. For more information, see Modify the scheduled restart time.