An ApsaraMQ for Kafka instance deployed in a single zone risks service unavailability and data loss during a zone-level failure. To protect against this, you can use the connector ecosystem integration feature of ApsaraMQ for Kafka to back up messages to a secondary instance in a different region. When a failure occurs, switch traffic to the secondary instance and restore service by resetting offsets.
How it works
This disaster recovery architecture uses two ApsaraMQ for Kafka instances deployed in different regions:
A primary instance receives and serves all production traffic.
A sink connector continuously backs up messages from the primary instance to a secondary instance in another region.
When a zone-level failure disrupts the primary instance, clients switch their endpoints to the secondary instance.
Consumers reset their offsets on the secondary instance and resume processing.
Prerequisites
Before you begin, make sure that you have:
Two ApsaraMQ for Kafka instances in different regions
Network connectivity between the two regions through Cloud Enterprise Network (CEN)
(Recommended) A custom domain name for CNAME-based traffic switching
Key considerations
Deploy the primary and secondary instances in different regions to protect against region-level failures.
After a failover, consumers reset offsets on the secondary instance and may reprocess some messages. Implement message idempotence in your consumers to handle duplicate consumption.
Use a CNAME record to map a custom domain name to the primary instance endpoint. During a failure, update the CNAME target to point to the secondary instance. This switches traffic without restarting your applications.
Set up disaster recovery
Step 1: Create a sink connector
Create an ApsaraMQ for Kafka sink connector to continuously replicate messages from the primary instance to the secondary instance.
For instructions, see Create ApsaraMQ for Kafka sink connectors.
Step 2: (Optional) Add a CNAME record
To enable fast traffic switching without application restarts, add a CNAME record that maps your custom domain name to the primary instance's domain name.
For instructions, see CNAME record.
Step 3: Configure the client endpoint
Choose one of the following approaches based on whether you set up a CNAME record:
CNAME mode (recommended)
Point your client to the custom domain name that has the CNAME record.
During a failure, update the CNAME target to the secondary instance's domain name. Traffic switches automatically without restarting your applications.
Regular mode
Point your client directly to the primary instance's endpoint.
During a failure, update the client configuration to the secondary instance's endpoint, then restart the application.
To access ApsaraMQ for Kafka instances across regions, connect the Virtual Private Clouds (VPCs) in those regions through Cloud Enterprise Network (CEN). For details, see Connect VPCs in different regions.