Data Transmission Service:Synchronize data from a PolarDB-X 1.0 instance to a DataHub project

Supported synchronization types

• Only incremental data synchronization and schema synchronization are supported. Full data synchronization is not supported.

• Only tables can be selected as synchronization objects. After a task starts, DTS does not synchronize columns added to the source database to the destination DataHub project.

Source database constraints

• Tables must have primary key or UNIQUE constraints, and all fields must be unique. Tables with only UNIQUE constraints do not support schema synchronization — use tables with primary key constraints. Tables with secondary indexes cannot be synchronized.

• If you select tables as synchronization objects and want to rename tables or columns in the destination, a single task supports up to 5,000 tables. For more than 5,000 tables, configure multiple tasks or synchronize the entire database.

• The storage type of the PolarDB-X 1.0 instance must be ApsaraDB RDS for MySQL (custom or purchased instances). PolarDB for MySQL is not supported.

• PolarDB-X 1.0 storage resources support only horizontal splitting (databases and tables). Vertical splitting is not supported.

• Read-only instances at the PolarDB-X 1.0 compute layer are not supported.

Binary log requirements

The following requirements apply to the binary logs of ApsaraDB RDS for MySQL instances attached to the PolarDB-X 1.0 instance:

• Binary logging must be enabled, and the binlog_row_image parameter must be set to full. Otherwise, the precheck fails and the task cannot start.

• For incremental data synchronization, binary logs must be retained for at least 24 hours.

• For full and incremental data synchronization, binary logs must be retained for at least seven days.

If binary logs are not retained for the required duration, DTS may fail to obtain them, which can cause task failure or data loss. After full data synchronization completes, you can reduce the retention period to more than 24 hours. Make sure that you set the retention period of binary logs in accordance with the preceding requirements. Otherwise, the service reliability and performance stated in the Service Level Agreement (SLA) of DTS cannot be achieved.

Operational restrictions

During data synchronization:

• Do not scale the source instance (including attached ApsaraDB RDS for MySQL instances) or change the distribution of physical databases for logical databases or tables.

• Do not migrate hot tables, change shard keys, or perform online DDL operations on the source instance. Doing so causes task failure or data inconsistency.

• Do not use gh-ost or pt-online-schema-change to perform DDL operations on synchronized objects.

• Write data to the destination database only through DTS. Using other tools to write to the destination can cause data inconsistency, and data loss may occur when DMS performs online DDL operations.

• If you change the network type of the PolarDB-X 1.0 instance during synchronization, update the network connection information in the task configuration afterward.

• Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours.

Step 1: Go to the Data Synchronization Tasks page

1. Log on to the Data Management (DMS) console.

2. In the top navigation bar, click Data + AI.

3. In the left-side navigation pane, choose DTS (DTS) > Data Synchronization.

The navigation steps may vary based on the DMS console mode and layout. For details, see Simple mode and Customize the layout and style of the DMS console. Alternatively, go directly to the Data Synchronization Tasks page in the new DTS console.

Step 2: Configure source and destination databases

1. Select the region where the data synchronization instance resides.

   In the new DTS console, select the region in the top navigation bar instead.

2. Click Create Task. In the wizard, configure the following parameters.

Task settings

Source database

Destination database

Step 3: Test connectivity and proceed

Click Test Connectivity and Proceed.

DTS automatically adds its server CIDR blocks to the whitelist of Alibaba Cloud database instances (such as ApsaraDB RDS for MySQL or ApsaraDB for MongoDB). For self-managed databases on Elastic Compute Service (ECS) instances, DTS adds its CIDR blocks to the ECS security group rules — make sure the ECS instance can reach the database. For databases on multiple ECS instances, manually add the DTS CIDR blocks to each instance's security group rules. For on-premises or third-party cloud databases, manually add the DTS CIDR blocks to the database whitelist. For the full list of CIDR blocks, see Add the CIDR blocks of DTS servers.

Step 4: Configure objects and settings

Basic settings

Parameter	Description
Synchronization Types	Incremental Data Synchronization is selected by default. You can also select Schema Synchronization only. Full Data Synchronization is not available for this source-destination combination. During schema synchronization, DTS synchronizes schemas of selected objects (such as tables) to the destination DataHub project.
Processing Mode of Conflicting Tables	Precheck and Report Errors (default): checks whether the destination has tables with the same names as the source. If identical names exist, the precheck fails and the task cannot start. To handle conflicts, use the object name mapping feature to rename tables in the destination. See Map object names. Ignore Errors and Proceed: skips the precheck for identical table names. Warning Selecting this option may cause data inconsistency. If schemas match and a record with the same primary key or unique key exists in the destination: during full synchronization, the destination record is kept; during incremental synchronization, the destination record is overwritten. If schemas differ, initialization may fail or only some columns may be synchronized.
Naming Rules of Additional Columns	During synchronization to a DataHub project, DTS adds extra columns to the destination topic. If these column names conflict with existing columns in the topic, synchronization fails. Set this to New Rule or Previous Rule based on your requirements. Before changing this setting, verify that additional columns and existing columns do not have name conflicts. See Modify the naming rules for additional columns.
Capitalization of Object Names in Destination Instance	The capitalization policy for database names, table names, and column names in the destination. Default: DTS default policy. See Specify the capitalization of object names in the destination instance.
Source Objects	Select one or more objects and click to move them to Selected Objects. Only tables can be selected. Selecting an entire database does not synchronize table creation or deletion changes to the destination.
Selected Objects	To rename a single object in the destination, right-click it. To rename multiple objects at once, click Batch Edit in the upper right. For details, see Map object names. To filter rows using WHERE conditions, right-click an object and specify the conditions. See Specify filter conditions.

Advanced settings

Step 5: Save settings and run the precheck

To preview the OpenAPI parameters for this task configuration, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

Click Next: Save Task Settings and Precheck.

DTS runs a precheck before the task can start. If the precheck fails, click View Details next to each failed item, resolve the issues, and run the precheck again. If a precheck alert can be ignored, click Confirm Alert Details > Ignore > OK, then click Precheck Again. Ignoring alerts may cause data inconsistency.

Step 6: Purchase the synchronization instance

1. Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the purchase page, configure the following parameters.

1. Read and select Data Transmission Service (Pay-as-you-go) Service Terms.

2. Click Buy and Start, then click OK in the confirmation dialog.

The task appears in the task list. When DTS synchronizes data from a PolarDB-X 1.0 instance, data is distributed across the attached ApsaraDB RDS for MySQL instances, and DTS runs a subtask for each instance. View the running state of each subtask in Task Topology.