Simulate node or zone failures on a PolarDB for MySQL cluster to verify that your high-availability architecture recovers as expected.
The disaster recovery drill feature is in canary release. To enable it for your cluster, contact us.
How it works
A disaster recovery drill injects a fault into a selected node or zone, makes the target unavailable, and then triggers the cluster's built-in recovery process. The drill runs in two dimensions:
| Dimension | Scope | Recovery target |
|---|---|---|
| Node | One node at a time | Hot standby node or read-only node is promoted; or a new read-only node is created |
| Zone | All nodes in the selected zone | Business switches over to the secondary zone |
Use the node dimension to test individual node failover. Use the zone dimension to test a full zone outage and cross-zone switchover.
Prerequisites
Before you begin, ensure that your cluster meets the following requirements:
Edition: Enterprise Edition
Engine version:
MySQL 5.6, revision version 5.6.1.0.42 or later
MySQL 5.7, revision version 5.7.1.0.26 or later
MySQL 8.0.1, revision version 8.0.1.1.30.2 or later
MySQL 8.0.2, revision version 8.0.2.2.19 or later
Risks and mitigations
Disaster recovery drills carry risks of transient connections and data loss. To avoid affecting your production environment, run drills on a cloned cluster. Pair drills with Intelligent Stress Testing to simulate realistic production workloads.
Data loss conditions:
| Drill dimension | Condition | Data loss |
|---|---|---|
| Zone | Recovery Point Objective (RPO) < 60 | May occur |
| Node | Cluster has hot standby nodes or read-only nodes, and RPO is 0 | No data loss |
Run a disaster recovery drill
Log on to the PolarDB console.
In the left-side navigation pane, click Clusters.
In the upper-left corner, select the region where the target cluster is deployed.
Find the cluster and click its ID.
In the left-side navigation pane, click Service Availability. On the page that appears, click the Drill tab.
On the Drill page, select the zone or nodes for the drill. Click Start Drill.
If you select a zone, the drill runs on all nodes in that zone. The Drill page also provides shortcuts to Intelligent Stress Testing and Clone Cluster.
In the Fault Injection Method dialog box, select Fault Injection By Node. Click OK.
On the Drill page, monitor the drill progress in the Drill List section. The list shows the current status and phase of the drill task.
Node dimension
Zone dimension
Drill phases
Fault injection
Faults are injected based on the Drill Nodes and Fault Injection Method values you selected.
Recovery
Node dimension
The recovery path depends on which node you selected and the cluster's current composition:
| Selected node | Cluster composition | Recovery action |
|---|---|---|
| Primary node | Has a hot standby node | Hot standby node is promoted to primary node |
| Primary node | Has a read-only node, no hot standby node | Read-only node is promoted to primary node |
| Primary node | No hot standby node or read-only node | Drill runs in zone dimension instead |
| Read-only node | Any | A new read-only node is created |
Zone dimension
All nodes in the selected zone are taken offline, and business switches over to the secondary zone.
The following figures show the architecture before, during, and after the zone-dimension drill.
Architecture before the drill:
All nodes in the zone taken offline:
Primary node becomes available in the secondary zone:
The system then determines whether to rebuild other nodes in the new primary zone based on available resources:
Post-drill
This phase applies only to zone-dimension drills. The duration depends on data volume and network latency. Business is not affected during this phase.
If other nodes were rebuilt in the new primary zone:
Rebuild read-only nodes:
Switch back primary node:
If other nodes were not rebuilt in the new primary zone:
Rebuild read-only nodes:
Switch back primary node:
What's next
Clone a cluster to create an isolated environment for future drills
Intelligent Stress Testing to simulate production workloads during drills