PolarDB:Synchronize data from a PolarDB for MySQL cluster to a self-managed Kafka cluster

Step 1: Purchase a DTS instance

Purchase a DTS instance. On the buy page, set:

• Source Instance: PolarDB

• Target Instance: Kafka

• Synchronization Topology: One-Way Synchronization

Step 2: Open the DTS console

Log on to the DTS console.

If you are redirected to the Data Management (DMS) console, click the icon in the to switch to the previous version of the DTS console.

Step 3: Configure the synchronization task

1. In the left-side navigation pane, click Data Synchronization.

2. On the Synchronization Tasks page, select the region where the destination Kafka cluster resides.

3. Find the instance you purchased and click Configure Task in the Actions column.

4. Configure the source and destination instances.

   

   Source instance

   Parameter	Description
   Instance type	Set to PolarDB Instance (fixed).
   Instance region	The source region selected on the buy page (read-only).
   PolarDB instance ID	Select the PolarDB for MySQL cluster ID.
   Database account	Enter the database account. The account must have read permissions on the objects to be synchronized.
   Database password	Enter the password for the database account.

   Destination instance

   Parameter	Description
   Instance type	Select the deployment type of your Kafka cluster. This example uses User-Created Database in ECS Instance. For other instance types, configure the network environment first. See Preparation overview.
   Instance region	The destination region selected on the buy page (read-only).
   ECS instance ID	Select the Elastic Compute Service (ECS) instance where the Kafka cluster is deployed.
   Database type	Select Kafka.
   Port number	Enter the Kafka service port. The default port is 9092.
   Database account	Enter the Kafka login username. Leave blank if the cluster has no authentication enabled.
   Database password	Enter the Kafka password. Leave blank if the cluster has no authentication enabled.
   Topic	Click Get Topic List and select a topic from the drop-down list.
   Kafka version	Select the version of the destination Kafka cluster.
   Encryption	Select Non-encrypted or SCRAM-SHA-256 based on your security requirements.

   Synchronization task name DTS generates a task name automatically. Specify a descriptive name for easy identification. The name does not need to be unique.

5. Click Set Whitelist and Next.

   DTS adds its server CIDR blocks to the whitelist of the source PolarDB cluster and to the inbound rules of the destination ECS instance so that DTS servers can reach both endpoints.

6. Select the objects to synchronize.

   Parameter	Description
   Data format in Kafka	Select Avro or Canal JSON. See Data formats of a Kafka cluster.
   Policy for shipping data to Kafka partitions	Select the partition routing policy based on your workload. See Specify the policy for synchronizing data to Kafka partitions.
   Objects to be synchronized	In the Available section, select one or more tables and click the icon to move them to the Selected section. DTS maps each table to the topic selected in the previous step. To change the topic a table maps to, use object name mapping. See Rename an object to be synchronized.
   Rename databases and tables	Use object name mapping to rename objects at the destination. See Object name mapping.
   Retry time for failed connections	Default is 720 minutes (12 hours). If DTS reconnects within this window, the task resumes automatically. Otherwise, the task fails. DTS charges apply during the retry period — set this value based on your requirements and release the DTS instance promptly if both source and destination instances are released.

   

7. Click Next.

8. Configure initial synchronization.

   Parameter	Description
   Initial synchronization	Select both Initial Schema Synchronization and Initial Full Data Synchronization. DTS synchronizes schemas and historical data first, then switches to incremental synchronization.
   Filter options	Ignore DDL in incremental synchronization phase is selected by default. DTS does not forward DDL operations from the source database during incremental synchronization.

   

9. Click Precheck. DTS runs a precheck before the task starts. If any item fails, click the icon next to the failed item to view details. Fix the issues and run the precheck again. If an item does not affect your use case, ignore it and rerun the precheck.

10. After the Precheck Passed message appears, close the dialog. The synchronization task starts automatically.