PolarDB:Disaster recovery drill

Overview

You can use the Disaster Recovery Drill feature to simulate failures on a PolarDB for MySQL cluster. A Disaster Recovery Drill supports two dimensions: the node dimension and the zone dimension. In a node-dimension drill, you can test only one node at a time. During the drill, once the system injects a fault, the specified node becomes unavailable, and the PolarDB for MySQL cluster recovers based on a fixed strategy. For more information about the recovery strategy, see Recovery phase.

Prerequisites

Your PolarDB for MySQL cluster must meet the following requirements:

• Edition: Enterprise Edition.

• Database engine:

  • MySQL 5.6, with a revision version of 5.6.1.0.42 or later.

  • MySQL 5.7, with a revision version of 5.7.1.0.26 or later.

  • MySQL 8.0.1, with a revision version of 8.0.1.1.30.2 or later.

  • MySQL 8.0.2, with a revision version of 8.0.2.2.19 or later.

Usage notes

A disaster recovery drill carries risks, such as transient connection interruptions and data loss. To avoid impacting your business, we strongly recommend running drills on a cloned cluster. You can also use Intelligent Stress Testing to simulate production workloads during a drill.

Procedure

1. Log on to the PolarDB console. In the left-side navigation pane, click Clusters, select the region where the cluster resides, and then click the ID of the target cluster to go to the cluster details page.

2. In the left-side navigation pane, click Service availability, and then click the Disaster Recovery Drill tab.

3. On the Disaster Recovery Drill page, select the zone or node for the drill. Then, click Start Drill.

   Note

   • The Disaster Recovery Drill page provides shortcuts to Intelligent Stress Testing and Clone Cluster.

   • If you select a zone, the drill runs on all nodes in that zone.

4. In the Fault Injection Method dialog box, select Fault Injection By Node. Click OK.

5. On the Disaster Recovery Drill page, you can view the status and phases of the current drill task in the Drills section at the bottom of the page.

   • Node-dimension schematic:

     • The drill list table contains columns such as Task ID, Drill Mode, Fault Injection Method, Start Time, End Time, Drill Status, and Node Information.

     • When you expand a task, you can view the Disaster Recovery Drill Phase Display, which contains three phases: Fault Injection (fault injection on the primary node), Recovery Phase (the primary node is restored and available), and Post-processing Phase. After each phase is completed, a green check mark is displayed.

   • Zone-dimension schematic:

     • The drill list table contains columns such as Task ID, Drill Mode, Fault Injection Method, Start Time, End Time, Drill Status, and Node Information.

     • When you expand a task, you can view the Disaster Recovery Drill Phase Display, which contains three phases: Fault Injection (fault injection on the read-only node and on the primary node), Recovery Phase (the primary node is restored and available), and Post-processing Phase (standby rebuild). After each phase is completed, a green check mark is displayed.

Drill phases