This document describes the nature, potential risks, and precautions of the disaster recovery drill feature. Read all terms carefully before initiating a drill. You can proceed only after selecting the confirmation checkbox.
-
Scope: Workspaces with same-city high availability (cross-zone deployment) enabled.
-
Scenario: Read this document before initiating a disaster recovery drill.
Drill mechanism
A disaster recovery drill simulates a failure in a specified availability zone (AZ) to verify the automatic migration and recovery capabilities of Flink jobs. During the drill, the system performs the following operations in sequence:
-
Set a zone blacklist: The selected target AZ is marked as "failed" to simulate its unavailability.
-
Trigger failure detection: After detecting that a node is unavailable, the system automatically starts resource rebalancing.
-
Pod eviction and reconstruction: The compute nodes (Pods) of the affected jobs are evicted from the failed AZ and rescheduled to healthy AZs.
-
Job state recovery: Jobs resume from the most recent successful checkpoint.
-
Remove blacklist and generate report: After the drill is complete, the normal AZ status is restored, and a drill result report is automatically generated.
Risk disclosure terms
By initiating a drill, you acknowledge and confirm the following:
Term 1: Brief job interruption
Affected jobs will be briefly interrupted during the drill. The duration depends on the job state size, checkpoint interval, and resource scheduling speed, and typically ranges from tens of seconds to several minutes. During this period, upstream and downstream data links (such as Kafka consumption and database writes) may experience brief data delays or backlogs.
Term 2: High availability of upstream and downstream systems must be ensured independently
A disaster recovery drill only verifies the automatic migration and recovery capabilities of Flink jobs in an AZ failure scenario. End-to-end business continuity depends on upstream and downstream systems (such as Kafka, ApsaraDB RDS, Hologres, Paimon, and OSS) also having cross-zone high availability capabilities. If upstream data sources or downstream write targets do not have cross-zone deployment or disaster recovery mechanisms enabled, migrated jobs may experience connection timeouts, write failures, or data backlogs. The Flink platform is not responsible for business interruptions caused by single-zone deployments of upstream or downstream systems.
Term 3: Best-effort recovery in hybrid billing mode
If a workspace uses hybrid billing (subscription + pay-as-you-go), jobs on elastic resources operate in best-effort recovery mode during the drill. If elastic resources in healthy AZs are insufficient, jobs on elastic resources may not be able to recover immediately and must wait for resources to be released or scaled out.
Term 4: Drill interval and frequency
-
Only one disaster recovery drill can be in progress at a time in a workspace.
-
Each drill has a 15-minute job migration window. Jobs that fail to migrate within this window are skipped, and the job recovery rate may fall below 100%. After the drill is complete, you must manually click End Drill to restore the target AZ.
Prerequisites
Confirm that all of the following conditions are met before initiating a drill:
|
Item |
Requirement |
|
Business period |
The current time is not during peak business hours. |
|
Job status |
All affected jobs are in the RUNNING state. |
|
Network connectivity |
The vSwitch CIDR blocks of all AZs involved in the drill (including the simulated failure AZ and healthy AZs) have been added to the access whitelists of upstream and downstream systems. |
|
Permissions |
The |
|
Checkpoint status |
Checkpointing is enabled for all affected jobs, and recent successful checkpoint records exist. If checkpointing is not enabled or no successful records exist, jobs cannot restore from state after the drill, which may cause duplicate data processing or data loss. |
Disclaimer
-
A disaster recovery drill is a simulated operation designed to verify high availability capabilities. Alibaba Cloud has implemented sufficient technical measures to ensure the safety and controllability of the drill process, but assumes no additional liability beyond the Service Level Agreement (SLA) for brief business interruptions, duplicate data processing, or delays caused by the drill.
-
Business losses resulting from initiating a drill during peak business hours, failing to confirm prerequisites, or failing to assess the impact of cross-zone latency are the responsibility of the user.
-
Drill results are affected by multiple factors (such as job state size, checkpoint configuration, and resource load). Historical drill results do not guarantee future drill performance.