Intra-region disaster recovery - Realtime Compute for Apache Flink

How it works

A Realtime Compute for Apache Flink workspace supports two types of compute resources that are measured in CUs: single-zone and cross-zone. These CU types correspond to single-zone and cross-zone namespaces. The following list describes the differences between the two types of namespaces:

In a single-zone namespace, Realtime Compute for Apache Flink allocates an optimal compute availability zone within your selected region. Availability zones within the same region are interconnected through an internal network. The compute layer transparently schedules resources across availability zones and pools inventory to improve resource elasticity and load balancing.
A cross-zone namespace provides cross-zone HA. If the availability zone where a job is running fails, the job automatically fails over to a stable availability zone in the same region. This design prevents service interruptions from a single availability zone failure and ensures the high availability of your jobs.

Basic concepts

Concept	Description
Zone	A physical area within a region where power and network are independent. Zones within a region are interconnected by a low-latency network.
RPO (Recovery Point Objective)	The maximum amount of data loss tolerated when a failure occurs. In Flink cross-zone mode, checkpoint state is persisted to zone-redundant OSS storage, so the RPO is aligned with that of OSS zone-redundant storage, which guarantees data integrity and consistency.
RTO (Recovery Time Objective)	The time required to recover the service after a failure. In Flink cross-zone mode, the RTO is at the minute level, covering JobManager switchover, resource allocation, state recovery, and data replay.
JobManager	The Flink control node responsible for job scheduling, checkpoint coordination, and resource management. In cross-zone deployments, it runs in Active-Standby mode.
TaskManager	The Flink compute node responsible for executing operator tasks and storing runtime state.
Checkpoint	The Flink periodic state snapshot mechanism that persists job state to external storage (OSS) for failure recovery.
State backend	The Flink state storage engine. Incremental state snapshots are written to remote storage at checkpoint time.

Failure recovery mechanism

Flink uses checkpoints and an automatic failure recovery mechanism to ensure that jobs recover automatically when a zone fails:

Checkpoint persistence: While a job is running, state snapshots are asynchronously written to OSS (zone-redundant storage) on a periodic basis, so state data is not lost when a zone fails.
JobManager active-standby switchover: When the active JobManager fails, the standby JobManager is elected as the new leader through ZooKeeper and takes over job management.
TaskManager rescheduling: After TaskManagers in the failed zone are lost, ResourceManager requests new TaskManagers in the surviving zones and reschedules the affected tasks.
State recovery: Newly scheduled TaskManagers restore state from the latest checkpoint. The RocksDB state backend supports incremental recovery, which shortens the recovery time.
Data replay: With a replayable data source (such as Kafka), consumption resumes from the offset recorded in the checkpoint, which preserves exactly-once semantics and ensures no data loss or duplication.

Failure scenarios and recovery

Single-zone failure

When a zone fails, Flink handles it as follows:

Failure detection: JobManager detects that TaskManager heartbeats from the failed zone have timed out and marks those nodes as unavailable.
JobManager switchover: If the active JobManager was in the failed zone, the standby JobManager is migrated and started up.
Task rescheduling: ResourceManager requests new TaskManager resources in the surviving zones and reschedules the affected tasks.
State recovery: The new TaskManagers restore job state from the latest checkpoint.
Data replay: The source resumes consuming from the checkpoint offset (for example, from Kafka) and processing continues.
Client transparency: The Flink job's sources and sinks automatically reconnect to upstream and downstream systems, so the recovery process is transparent to them.

Recovery metrics:

Metric	Value
RPO (data loss)	0 (Checkpoint state is persisted to zone-redundant OSS storage, so the RPO is aligned with that of OSS zone-redundant storage, which guarantees data integrity and consistency.)
RTO (service recovery time)	Minute level (1 to 10 minutes, depending on state size and checkpoint interval)

Usage notes

End-to-end high availability requires your upstream and downstream systems to be highly available as well.
When Realtime Compute for Apache Flink accesses external upstream and downstream systems across zones, it may introduce millisecond-level latency. The service-level agreement (SLA) guarantees that network latency between availability zones is within 3 ms. For more information, see Average network latency between availability zones in a region. The latency is generally acceptable for most upstream and downstream systems, except for Tair. We recommend that you evaluate your business's latency requirements to determine whether to enable cross-zone HA.
If you enable cross-zone HA for a workspace with hybrid billing, the system prioritizes the migration and recovery of jobs on Subscription resources during a failure. For jobs on pay-as-you-go compute resources, the system makes a best-effort recovery attempt at no extra charge.

Regions and billing

Currently, only workspaces that use an Intel X86 architecture and the Subscription billing method (including hybrid billing) support the purchase of cross-zone compute resources. These resources are priced higher than single-zone resources. For more information about supported regions and pricing, see Subscription.

If your workspace uses the pay-as-you-go billing method, you must first switch to Subscription, and then refer to Method 2 and Method 3.

Note

You cannot change workspaces with the cross-zone deployment model to the pay-as-you-go billing method.

Enable and use cross-zone HA

Method 1: Create a cross-zone namespace at workspace creation

You can automatically create a cross-zone namespace when creating a Realtime Compute for Apache Flink workspace.

For a workspace that uses an Intel X86 architecture and the Billing Method billing method, set the deployment model to cross-zone, specify the Resource Quota for the cross-zone CU type, and select vSwitches from two different availability zones. For more information, see Activate Realtime Compute for Apache Flink.

After you create the workspace, the system automatically generates a cross-zone namespace named workspace-name-default-ha.
Click the name of the target cross-zone namespace to create and run jobs in it.

In the namespace list, you can identify a cross-zone namespace by the CU Type column, which displays cross-zone. Click the namespace name to enter it.

Method 2: Add cross-zone CUs and create a namespace

Use this method if the deployment model of your workspace was not set to cross-zone when you created the workspace. You can add cross-zone CUs and then create a cross-zone namespace.

Add cross-zone CUs.
1. In the Actions column of the target workspace, choose More > Reconfigure Workspace Resources.
2. Click Add CU Type.
3. Enter the target number of CUs for the cross-zone type and select the vSwitches.
  
  Only existing vSwitches in the target VPC and in availability zones supported by Realtime Compute for Apache Flink are displayed. If no vSwitch is available, you can create a vSwitch.
  
  Important
  After you add cross-zone CUs to the workspace, make sure to add the CIDR block of the vSwitch in the new availability zone to the allowlists of your upstream and downstream systems to prevent network access issues.
  
  Read and agree to the terms of service, and then click OK.
4. Select the agreement and click OK to complete the payment.
Add a cross-zone namespace. For more information, see Add a namespace.
Create and run jobs in the cross-zone namespace. For more information, see Enter a namespace.

Method 3: Enable cross-zone HA for a namespace

Log on to the Realtime Compute for Apache Flink console.
Confirm that the workspace has enough unallocated cross-zone CUs to match the CUs allocated to the target namespace.

If the number of unallocated cross-zone CUs is insufficient, you can add more cross-zone CUs to the workspace or release surplus cross-zone CUs from other cross-zone namespaces by clicking Reconfigure Resources. For more information, see Adjust resources.

In the Actions column of the target namespace, click Turn on high availability, and then click OK.

After you click OK, the system replaces the single-zone CUs in the namespace with an equivalent number of cross-zone CUs and returns the original single-zone CUs to the workspace. You can then reallocate these CUs or scale down your resources to reduce costs.

For example, assume that Workspace A has 5 single-zone CUs and 5 cross-zone CUs. Namespace A is allocated 2 single-zone CUs. After you enable cross-zone HA for Namespace A, its 2 allocated single-zone CUs are returned to the unallocated single-zone resource pool of Workspace A. Then, 2 cross-zone CUs are allocated to Namespace A. The following table compares the resource status of the workspace before and after you enable cross-zone HA.

Status	Single-zone CUs	Cross-zone CUs
Before enabling	2/5	0/5
After enabling	0/5	2/5

Create and run jobs in the converted cross-zone namespace. For more information, see Enter a namespace.

Disable cross-zone HA for a namespace

Regardless of how you created the cross-zone namespace, follow these steps to disable the feature. However, disabling cross-zone HA does not stop billing. To stop billing, see Unsubscribe from cross-zone CUs.

Log on to the Realtime Compute for Apache Flink console.
Confirm that the workspace has enough unallocated single-zone CUs to match the CUs allocated to the target cross-zone namespace.

If the number of unallocated single-zone CUs is insufficient, you can add more single-zone CUs to the workspace or release surplus single-zone CUs from other single-zone namespaces by clicking Reconfigure Resources. For more information, see Adjust resources.
In the Actions column of the target namespace, click Turn off high availability, and then click OK.

After you disable cross-zone HA, the resource changes are the reverse of the enabling process. You can resume normal use of the namespace only after the process is complete.

Unsubscribe from cross-zone CUs

You will not be charged cross-zone fees when the number of Allocated cross-zone CUs for your workspace is 0. To unsubscribe from cross-zone CUs, perform the following steps:

Log on to the Realtime Compute for Apache Flink console.
Confirm that cross-zone HA is disabled for all namespaces in the workspace.

For more information, see Disable cross-zone HA for a namespace.
Set the target number of cross-zone CUs in the workspace to 0.
1. In the Actions column of the target workspace, choose More > Reconfigure Workspace Resources.
2. Change the Target CU value for the cross-zone CU type to 0.
3. Agree to the terms and click OK.

Realtime Compute for Apache Flink:Cross-zone high availability