Disaster recovery drill (Beta) - Realtime Compute for Apache Flink

Overview

A disaster recovery drill is initiated at the workspace level. Any workspace that contains a cross-zone (same-city high availability) namespace can run the drill. During a drill, the system adds the target AZ to the zone blacklist, triggers Flink failure detection and resource rebalancing, evicts affected jobs, reschedules them in healthy AZs, and restores their state from the most recent successful checkpoint. After the drill ends, the system generates an HTML drill report for audit records.

The drill feature provides self-service verification and compliance audit support for high-availability customers, without manual intervention from platform SREs.

Limits

Only Intel x86 architecture is supported. ARM architecture is not supported due to differences in the underlying image.
The workspace must contain a cross-zone namespace, that is, same-city high availability must be enabled. For instructions, see Same-city high availability.
While a drill is in progress, scaling operations are prohibited on the target workspace, and deletion and scaling operations are prohibited on the involved namespaces.
In hybrid billing mode, jobs on elastic resources are recovered on a best-effort basis. If elastic resources in healthy AZs are insufficient, elastic job recovery may be delayed.

Prerequisites

A cross-zone namespace with same-city high availability is purchased and enabled.
The vSwitch CIDR blocks of all AZs involved in the drill (including the simulated failure AZ and the healthy AZs) are added to the access whitelists of upstream and downstream systems, such as Kafka, ApsaraDB RDS, Hologres, Paimon, and OSS.
All affected jobs are in the RUNNING state, checkpointing is enabled, and successful checkpoint records exist.
The operator account has the flink:DescribeDisasterRecoveryDrill, flink:DescribeInstanceZoneJobs, flink:ListDisasterRecoveryDrills, flink:StartDisasterRecoveryDrill, and flink:StopDisasterRecoveryDrill permissions.

Apply for the disaster recovery drill whitelist

The disaster recovery drill is a whitelist feature. You must apply for access before you can use it.

Log on to the Realtime Compute for Apache Flink console.
The namespace list of the target workspace contains a cross-zone namespace.
In the Actions column of the workspace, click Disaster Recovery Drill > Start Drill.

Note

Single-zone namespaces do not display the Disaster Recovery Drill menu. If the Disaster Recovery Drill entry is grayed out, hover over it to view the reason, such as "Whitelist verification failed" or "Same-city high availability not enabled".

Initiate a disaster recovery drill

Important

Initiate the drill during off-peak hours. Affected jobs are briefly interrupted during the drill, typically for tens of seconds to several minutes.
End-to-end business continuity also relies on upstream and downstream systems having cross-zone high availability. If they are not deployed across zones, migrated jobs may experience connection timeouts, write failures, or data backlogs. For risk terms, see Informed consent for high-risk operations in disaster recovery drills.

Step 1: Select the target failure AZ

Log on to the Realtime Compute for Apache Flink console.
In the Actions column of the target workspace, click Disaster Recovery Drill > Start Drill.
In the Target Failure AZ drop-down list, select the AZ to simulate as failed.

After you select a failure AZ, the affected jobs are displayed.

After you confirm the AZ selection, proceed to Step 2.

Step 2: Confirm the impact scope

Review the Impact Scope Estimation and confirm the following information:

Item	Description
Affected jobs	The number of jobs in the RUNNING state on the target failure AZ.
RTO	The estimated time to recover from the failure AZ to a healthy AZ.
Affected job list	The name, current status, and AZ of each affected job.

Select the following three confirmation checkboxes:
- The current time is not during peak business hours.
- I acknowledge that jobs may be briefly interrupted during the drill.
- I have read the Informed Consent for High-Risk Operations.

Important

Read all terms in the Informed Consent for High-Risk Operations, covering brief job interruptions, self-guaranteed upstream and downstream high availability, best-effort recovery under hybrid billing, and drill interval and frequency.
Verify the Drill Summary and confirm the target failure AZ, number of affected jobs, and estimated RTO.

Monitor drill progress

After the drill starts, the page automatically redirects to the drill progress page, which displays the entire drill process in real time.

Page elements:

Element	Description
Progress bar	Displays the current step, total steps, elapsed time, and progress percentage.
AZ topology diagram	Healthy AZs are shown in green. The failure AZ is shown in red and labeled "Simulating failure". Red arrows indicate the job migration direction.
Drill step timeline	Displays the execution status of five steps in sequence: simulate AZ failure, system failure detection, execute failure migration, evaluate job impact, and generate drill report.

Important

The job migration window is 15 minutes. Jobs that fail to migrate within the window are skipped, and the job recovery rate may fall below 100%.
To terminate the drill early, click Stop in the lower-right corner and confirm in the secondary confirmation dialog box.
Whether the migration window expires, the drill is terminated early, or all jobs complete migration, you must manually click End Drill to remove the target AZ from the blacklist and resume scheduling.
Completed migration records are retained in the drill report. For jobs that fail to migrate within the window, the Checkpoint column in the job migration details shows Recovery failed. Core metrics are calculated based on the completed portion.

View drill results and download the report

After the drill migration ends, the page redirects to the drill results page.

Core information on the page:

Item	Description
Core metrics	Displays metrics such as RTO and job recovery rate.
Job migration details	Displays the original AZ, migration target AZ, migration duration, and checkpoint recovery status for each job.
Timeline	Records all events and timestamps of the drill process.

Click Download Report to download the drill audit report. The report contains the drill basic information, drill configuration, drill process timeline, and job migration details.

Note

Ending the drill adds the failure AZ back to the whitelist and restores it as a healthy AZ. Migrated jobs remain in the new AZ and are not automatically migrated back.

View drill history

In the Actions column of the target workspace, click Disaster Recovery Drill > Drill History.
View the last drill time, total number of drills, and the most recent result.
In the drill history table, click View Details to open the drill results page, and click Download Report to download the corresponding HTML drill report.

References

For how to enable and manage cross-zone namespaces, see Same-city high availability.
For drill risk terms and prerequisite confirmations, see Informed consent for high-risk operations in disaster recovery drills.
For namespace resource allocation and adjustment, see Adjust resources.