Data Transmission Service:Synchronize data from a PolarDB for MySQL cluster to a self-managed Doris database

Source database

• Tables with PRIMARY KEY or UNIQUE constraints: all fields must be unique, or the destination may contain duplicate data.

• Tables without PRIMARY KEY or UNIQUE constraints: select Schema Synchronization for Synchronization Types and duplicate for Engine in the Configurations for Databases, Tables, and Columns step.

• If you select tables (not databases) as synchronization objects and need to rename tables or columns in the destination, a single task supports up to 1,000 tables. Exceeding this limit causes a request error. Configure multiple tasks or sync the entire database instead.

• Binary log requirements:

  • loose_polar_log_bin must be set to ON

  • Retention period: at least three days, seven days recommended

• Do not run DDL operations that change database or table schemas during initial schema synchronization or initial full data synchronization. DTS queries the source database during full sync, which creates metadata locks that may block DDL operations.

• Changes from operations not recorded in binary logs—such as data restored from physical backups or generated by cascade operations—are not synced to the destination. If this occurs, you can remove and re-add the affected database or table from the synchronization objects if your business permits. See Modify synchronization objects.

General limitations

• DTS does not sync read-only nodes of the source PolarDB for MySQL cluster.

• DTS does not sync Object Storage Service (OSS) external tables from the source.

• Primary/standby switchover is not supported during initial full data synchronization. If a switchover occurs, reconfigure the synchronization task.

• Only tables using the Unique Key Model or Duplicate Key Model in Doris are supported. When using the Duplicate Key Model, DTS converts UPDATE and DELETE statements to INSERT statements. Duplicate data may appear if any of the following occur: Use the additional columns _is_deleted, _version, and _record_id to deduplicate if needed (see Additional columns added by DTS).

  • A retry operation runs on the data synchronization instance

  • The instance is restarted

  • Two or more DML operations are applied to the same row after the instance starts

• Only the bucket_count parameter can be specified in the Selected Objects section. The value must be a positive integer. Default: auto.

• Do not create clusters in the destination Doris database during synchronization. If this happens, the task fails. Restart the data synchronization instance to resume.

• Doris only supports database and table names that start with a letter. Use the object name mapping feature to rename any object whose name starts with a non-letter character.

• If a database, table, or column name contains Chinese characters, use object name mapping to rename it (for example, to an English name). Otherwise, the task may fail.

• You cannot modify DDL operations on multiple columns at once, or modify DDL operations on a table consecutively.

• Do not add backend (BE) nodes to the destination Doris database during synchronization. If this happens, the task fails. Restart the data synchronization instance to resume.

• In multi-table merge scenarios (multiple source tables syncing to one destination table), the schemas of all source tables must be identical. Otherwise, data inconsistency or task failure may occur.

• In PolarDB for MySQL, VARCHAR(M) uses character length for M. In Doris, VARCHAR(N) uses byte length for N. If you are not using schema synchronization, set the Doris VARCHAR length to four times the corresponding MySQL VARCHAR length to avoid data loss.

• When using DMS or gh-ost to perform online DDL changes on the source, DTS syncs only the original DDL to the destination. Online DDL changes made with pt-online-schema-change are not supported and may cause data loss or synchronization failure.

• During initial full data synchronization, DTS uses read and write resources of both databases. Evaluate the performance impact beforehand and schedule synchronization during off-peak hours, when CPU usage on both ends is below 30%.

• Concurrent INSERT operations during initial full data synchronization cause table fragmentation in the destination. After full sync completes, destination tables may occupy more storage than the source.

• Do not use pt-online-schema-change or similar tools for online DDL operations on synchronization objects during synchronization. This causes sync failure.

• If data is written to the destination database from sources other than DTS during synchronization, data inconsistency may occur between source and destination.

• DTS periodically executes CREATE DATABASE IF NOT EXISTS \test\`` on the source to advance the binary log position.

• During incremental synchronization, DTS uses a batch strategy and writes to each synchronization object at most once every five seconds. This may introduce a normal synchronization latency of less than 10 seconds. To reduce latency, adjust the selectdb.reservoir.timeout.milliseconds parameter (range: 1,000–10,000 milliseconds) in the DTS console. Shorter batching intervals increase write frequency, which may raise the destination's load and response time.

• If an instance fails, DTS support attempts recovery within eight hours. During recovery, the instance may be restarted or its parameters adjusted. Only DTS instance parameters are modified—database parameters remain unchanged.

Step 1: Go to the data synchronization page

Use one of the following methods:

Step 2: Configure source and destination databases

1. Click Create Task.

2. Configure the Task Name. DTS generates a name automatically. Specify a descriptive name to make the task easy to identify—a unique name is not required.

3. Configure the source and destination database connections using the parameters below.

Source database (PolarDB for MySQL)

Destination database (Doris)

1. Click Test Connectivity and Proceed. In the CIDR Blocks of DTS Servers dialog box, click Test Connectivity.

Make sure that the CIDR blocks of DTS servers are added to the security settings of both source and destination databases. See Add DTS server IP addresses to a whitelist.

Step 3: Select synchronization objects

1. In the Configure Objects step, configure the following:

1. Click Next: Advanced Settings and configure the following:

1. (Optional) Click Next: Configure Database and Table Fields. Specify Primary Key Column, Distribution Key, and Engine for each table.

This step is available only when Schema Synchronization is selected. Set Definition Status to All to view all tables.

Primary Key Column supports multiple columns. Distribution Key must be a subset of the primary key columns.

For tables without primary keys or UNIQUE constraints, select duplicate for Engine. Otherwise, the task may fail or data loss may occur.

Step 4: Run the precheck

Click Next: Save Task Settings and Precheck.

DTS runs a precheck before starting the task. The task starts only after passing all checks.

If any check fails, click View Details to see the cause, fix the issue, and rerun the precheck.

If a warning appears: fix and rerun if the item cannot be ignored; click Confirm Alert Details > Ignore > OK > Precheck Again if the item can be safely ignored. Ignoring alerts may cause data inconsistency.

To preview the OpenAPI parameters for this task configuration, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

Step 5: Purchase an instance

1. Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the purchase page, configure the following:

1. Read and select Data Transmission Service (Pay-as-you-go) Service Terms.

2. Click Buy and Start, then click OK in the dialog box.

The task appears in the task list. Monitor its progress from there.