Proactive O&M events

Overview

A proactive O&M event is a planned maintenance operation that SelectDB for Alibaba Cloud performs to improve service quality, mitigate potential risks, or apply necessary upgrades. These events are primarily categorized into version upgrades, deployment architecture upgrades, and ECS restarts. These events are essential for the long-term stability, security, and efficiency of your instance.

To minimize the impact on your business:

Schedule proactive O&M events during off-peak business hours.
Ensure your application has an automatic reconnection and retry mechanism.

Event types and business impact

Proactive O&M events are divided into the following three categories. Understanding the impact of each event type helps you better plan for business continuity.

Version upgrade

This event involves minor version upgrades, security fixes, or feature optimizations for the SelectDB kernel, proxy, or related components.

Typical scenarios
- Minor kernel version upgrades.
- Upgrades to the proxy or access layer components.
- Security-related updates, such as SSL certificate updates.
- Upgrades to monitoring, backup, and other dependent components.
Primary impact
- Transient connection interruption and read-only state: Upgrades typically involve an instance restart or switchover, which causes a brief connection interruption. To ensure data consistency, the instance may enter a read-only or unavailable state for a period ranging from seconds to a few minutes.
- Limited control plane operations: During an upgrade, control plane operations such as scaling and parameter modification are temporarily unavailable.
Key recommendations
- Your application must implement an automatic reconnection mechanism and a reasonable retry mechanism.
- Avoid running long-running transactions or large-scale data import and export jobs during the maintenance window.

Deployment architecture upgrade

This event involves optimizing or adjusting the underlying infrastructure, network, or high availability architecture.

Typical scenarios
- Replacement of the underlying host or node migration.
- Network topology adjustments or hardware upgrades.
- Changes to the high availability architecture, such as adjustments to the primary/standby setup or cross-availability zone deployment.
- Upgrade of the FE node deployment method, such as from Master/Observer to 3 Followers.
Primary impact
- Transient connection interruption and read-only state: An instance switchover causes a transient connection interruption and a brief read-only state.
- Impact on peripheral systems: If you use BI tools or data synchronization services, you may need to reconfigure the data source or restart synchronization tasks after the upgrade to rediscover and connect to the instance.
Key recommendations
- Strongly recommended: Always use the platform-provided domain name or connection string to access your instance. Avoid hard-coding any IP address in your application.
- Prepare contingency plans, such as data validation and traffic control, for the maintenance window.

ECS restart

The ECS where the instance is hosted restarts due to host maintenance or a specification change.

Typical scenarios
- Change in ECS instance specifications (CPU/memory).
- Host operating system patching or security hardening.
- Host hardware maintenance or exception fixing.
Primary impact
- Transient connection interruption and read-only state: The impact is similar to a version upgrade, causing a transient connection interruption and a brief read-only state.
- Long-running transactions or batch processing tasks that depend on a single node may be interrupted.
Key recommendations
- Treat this type of event as minor maintenance and ensure your application is fault-tolerant and has a retry mechanism.
- Closely monitor key business metrics, such as connection counts and error rates, before and after the maintenance window.

Event notifications

SelectDB for Alibaba Cloud provides multi-channel notifications and flexible time management to help you stay informed and properly schedule maintenance plans.

Event levels

Proactive O&M events are categorized by urgency and communicated through multiple channels.

Level	Description	Notification time	Management policy
S0 (Urgent) (Risk Fix)	Addresses critical vulnerabilities, security risks, or host anomalies that require immediate fixing to prevent major failures.	Notifications are typically sent at least 3 days before execution.	Cancellation is generally not supported. You can only make minor adjustments to the execution time within a limited range.
S1 (Scheduled) (System Maintenance)	Routine maintenance, such as feature optimizations, software or hardware upgrades, and architecture improvements.	Notifications are typically sent at least 3 days before execution.	You can generally adjust the scheduled switch time. In some cases, you can also cancel the scheduled event.

Notification channels

To receive timely event information, monitor the following channels:

Event Center in the console

The Event Center page in the SelectDB for Alibaba Cloud console provides detailed information about each event. This includes the event's type, affected instances, reason, impact, and current status (Scheduled, Completed, or Canceled).

Message Center (SMS messages, emails, and internal messages)

To avoid missing important notifications, go to Message Center and configure the following settings under Message Reception Management > Basic Reception Management > Cloud Database Failure or O&M Notification:

Select the SMS message and email notification methods.
Set the recipient to the actual database or application administrator.

Instructions

View and manage proactive O&M events

Log on to the SelectDB for Alibaba Cloud console.
In the left-side navigation pane, go to Event Center.
Above the list of scheduled events, click Scheduled to view all pending proactive O&M events.
For some S1 (Scheduled) events, you can select one or more events and then click Cancel Scheduled Configuration to cancel them.

Adjust execution time

SelectDB for Alibaba Cloud provides a flexible time management mechanism to help you control the impact of maintenance within an expected range. The execution time is determined based on the following order of priority: scheduled switch time > periodic time > maintenance window.

Scheduled switch time (Per-event configuration, highest priority): This is a specific execution time for a single event. You can set a precise execution time for an event in the console before the Latest Operation Time.
Periodic time (Global maintenance window, medium priority): You can set a global, recurring maintenance window, such as every Tuesday from 00:00 to 02:00. For S1 (Scheduled) level events that do not have a specific scheduled switch time, the system automatically schedules them to run within this window.

Note
This setting may not apply to urgent S0 (Urgent) events.
Maintenance window (Fallback mechanism, lowest priority): This is the final fallback strategy at the instance level. If an event does not have a scheduled switch time and is not executed within the periodic time, the system will execute it during the maintenance window you set for the instance. Configure the maintenance window for all critical instances to be during off-peak business hours.

Important

All time adjustments must be completed before the Latest Operation Time indicated by the system.

Modify scheduled switch time

Go to Event Center.
Above the list of scheduled events, click Scheduled to view all pending proactive O&M events.
Select one or more events, and then click Configure Scheduled Time at the bottom of the list. In the pane that appears on the right, you can:
- Set a specific Scheduled switchover date and time.
- Or, select Set Earliest Execution Time, and the system will automatically schedule the execution after the time you specify.

Configure periodic time

On the Event Center page, click Configure Recurring Time.
Set your desired fixed maintenance window on a weekly or monthly basis.
After you save the configuration, subsequent S1 (Scheduled) events without a specified scheduled switch time will run within this window by default.

Modify maintenance window

See Set a maintenance window.

View historical events

Go to Event Center.
In the list of scheduled events, click Completed or Canceled.

You can view records of all completed or canceled events for change auditing and issue tracking.

Best practices and FAQ

Best practices

Design for high availability: Always use a domain name or connection string to access your instance, and ensure your application has a reliable automatic reconnection and retry mechanism.
Plan your maintenance time: Configure a maintenance window during off-peak hours for critical instances, and use the periodic time feature to centralize routine maintenance in a fixed window.
Configure O&M notifications: Ensure that notifications are correctly configured in Message Center and that the actual database or application administrator is added as a contact.

FAQ

Q: Are proactive O&M events executed automatically? Can I skip them?

A: Yes. To ensure service stability and security, events are executed automatically at the scheduled time. For S1 (Scheduled) events, you can usually adjust the execution time before the Latest Operation Time, and some events can be canceled. For S0 (Urgent) events, cancellation is typically not supported to ensure risks are addressed promptly.

Q: How much will a proactive O&M event impact my business?

A: Most events cause only a transient connection interruption and a brief read-only state, lasting from seconds to a few minutes. For online applications designed with an automatic reconnection mechanism, the impact is usually minimal. If your business uses long-lived connections or long-running transactions, pause these tasks before the maintenance window or ensure they are fault-tolerant and can resume from the point of interruption.