Data Transmission Service (DTS) lets you synchronize data from a PolarDB-X 2.0 instance to a PolarDB-X 1.0 instance, supporting both full and incremental data synchronization.
This feature is in beta testing and is available only to some users.
In this topic, you learn how to:
Review prerequisites and limitations before you begin
Grant the required permissions to database accounts
Configure a DTS synchronization task
Run a precheck and purchase the synchronization instance
Prerequisites
Before you begin, ensure that you have:
A destination PolarDB-X 1.0 instance with storage space larger than the data to be synchronized from the source PolarDB-X 2.0 instance. See Create a PolarDB-X 1.0 instance.
A database and tables created in the destination instance to receive the synchronized data. See Create a databaseCreate a database and Perform basic SQL operations.
Match the database and table names in the destination instance to those in the source PolarDB-X 2.0 instance. If the names differ, use the object name mapping feature in the Selected Objects box on the Configure Objects page when you configure the task.
Billing
| Synchronization type | Fee |
|---|---|
| Full data synchronization | Free of charge |
| Incremental data synchronization | Charged. See Billing overview. |
Limitations
Source database limits
Bandwidth: The server hosting the source database must have sufficient outbound bandwidth. Insufficient bandwidth reduces synchronization speed.
Read-only instances: Read-only instances of Enterprise Edition PolarDB-X 2.0 cannot be used as the source.
Table naming restrictions:
Tables with uppercase letters in their names cannot be synchronized.
Tables with reserved words as names (for example,
select) cannot be synchronized.TABLEGROUPobjects and databases or schemas that contain theLocalityattribute are not supported.
Primary key requirement: Tables must have
PRIMARY KEYorUNIQUEconstraints with all fields unique. Otherwise, duplicate records may appear in the destination database.Table count limit: If you select tables as sync objects and need to rename tables or columns in the destination, a single task supports up to 5,000 tables. Tasks exceeding this limit return a request error. Split the work across multiple tasks or sync at the database level instead.
Binlog settings: Binary logging is enabled by default on PolarDB-X 2.0. Set the
binlog_row_imageparameter tofull. If this parameter is not set tofull, the precheck fails and the task cannot start. See Parameter settings.DDL operations during full sync: Do not run DDL operations that change the database or table schema during full data synchronization. DDL operations can cause the task to fail.
During full data synchronization, DTS queries the source database. This creates metadata locks that may block DDL operations in the source database.
Network changes: If the network configuration of the PolarDB-X 2.0 instance changes, the synchronization instance may experience temporary latency.
Other limits
Heartbeat latency: DTS periodically runs
CREATE DATABASE IF NOT EXISTS \`test\`on the source database to write heartbeat data and advance the Binlog offset. If the Whether to delete SQL operations on heartbeat tables of forward and reverse tasks option is set to Yesalert notification settings (or if the database account used by the synchronization instance lacks the permission to create databases), and no DML operations run on the source database for an extended period, the displayed latency may be inaccurate.To refresh the latency, run a DML operation on the source database.
Database load during full sync: Full data synchronization consumes read and write resources on both the source and destination databases. Evaluate database performance before you start, and run synchronization during off-peak hours when the CPU load on both databases is below 30%.
Storage fragmentation: During full initialization, DTS runs concurrent INSERT operations, which can cause table fragmentation in the destination database. After full initialization completes, the storage space used by the destination tables is larger than that used by the source tables.
External writes: If data is written to the destination database from sources other than DTS while the synchronization instance is running, data inconsistency may occur between the source and destination databases, and the synchronization instance may fail.
Instance recovery: If an instance fails, DTS helpdesk will attempt to recover it within 8 hours. Recovery may involve restarting the instance or adjusting parameters.
Only DTS instance parameters are modified during recovery. Database parameters are not changed. For the list of parameters that may be modified, see Modify instance parameters.
SQL operations supported for incremental synchronization
| Operation type | SQL statements |
|---|---|
| DML | INSERT, UPDATE, DELETE |
| DDL | CREATE TABLE, RENAME TABLE, ALTER TABLE, TRUNCATE TABLE, DROP TABLE, CREATE INDEX, DROP INDEX |
Notes on DDL synchronization:
Only CREATE TABLE operations for single tables are supported. CREATE TABLE for sharded databases and tables is not supported.
If the source is an Enterprise Edition PolarDB-X 2.0 instance, CREATE INDEX operations cannot be synchronized.
RENAME TABLE may cause data inconsistency. If you select a table as a sync object and rename it during synchronization, data for that table stops being synchronized to the destination. To prevent this, add the database containing the table to the sync objects list — include both the pre-rename and post-rename database.
Permissions required for database accounts
| Database | Required permissions | Reference |
|---|---|---|
| Source PolarDB-X 2.0 instance | REPLICATION SLAVE, REPLICATION CLIENT, and SELECT on the objects to synchronize | Manage database accounts and Account permission issues during data synchronization |
| Destination PolarDB-X 1.0 instance | Read and write permissions | Manage database accounts |
Create a synchronization task
Step 1: Go to the Data Synchronization page
Use one of the following methods to open the Data Synchronization page.
DTS console
Log on to the DTS console.DTS console
In the left-side navigation pane, click Data Synchronization.
In the upper-left corner, select the region where the synchronization task resides.
DMS console
The steps below may vary based on your DMS console mode and layout. See Simple mode and Customize the layout and style of the DMS console.
Log on to the DMS console.DMS console
In the top navigation bar, move the pointer over Data + AI and choose DTS (DTS) > Data Synchronization.
From the drop-down list next to Data Synchronization Tasks, select the region where the synchronization instance resides.
Step 2: Configure source and destination databases
Click Create Task to open the task configuration page.
Configure the source and destination databases using the parameters below.
Source database
| Parameter | Description |
|---|---|
| Task Name | The name of the DTS task. DTS generates a name automatically. Specify a descriptive name to make the task easy to identify. The name does not need to be unique. |
| Select Existing Connection | If you have a database instance registered with DTS, select it from the drop-down list. DTS automatically fills in the connection parameters. If you are not using a registered instance, enter the connection details manually. In the DMS console, select from the Select a DMS database instance drop-down list. |
| Database Type | Select PolarDB-X 2.0. |
| Access Method | Select Alibaba Cloud Instance. |
| Instance Region | Select the region where the source PolarDB-X 2.0 instance resides. |
| Replicate Data Across Alibaba Cloud Accounts | Select No to use an instance in the current Alibaba Cloud account. |
| Instance ID | Select the ID of the source PolarDB-X 2.0 instance. |
| Database Account | Enter the database account for the source instance. For permission requirements, see Permissions required for database accounts. |
| Database Password | Enter the password for the database account. |
Destination database
| Parameter | Description |
|---|---|
| Select Existing Connection | If you have a database instance registered with DTS, select it from the drop-down list. DTS automatically fills in the connection parameters. If you are not using a registered instance, enter the connection details manually. In the DMS console, select from the Select a DMS database instance drop-down list. |
| Database Type | Select PolarDB-X 1.0. |
| Access Method | Select Alibaba Cloud Instance. |
| Instance Region | Select the region where the destination PolarDB-X 1.0 instance resides. |
| Instance ID | Select the ID of the destination PolarDB-X 1.0 instance. |
| Database Account | Enter the database account for the destination instance. For permission requirements, see Permissions required for database accounts. |
| Database Password | Enter the password for the database account. |
Click Test Connectivity and Proceed.
Make sure DTS server CIDR blocks are added to the security settings of the source and destination databases. See Add DTS server IP addresses to a whitelist.
Step 3: Configure sync objects
On the Configure Objects page, configure the following settings.
| Parameter | Description |
|---|---|
| Synchronization Types | Incremental Data Synchronization is selected by default. To also sync existing data, add Full Data Synchronization. Schema Synchronization is not available for this source-destination pair. When full data synchronization is enabled, DTS copies the historical data from the source to the destination as the baseline for subsequent incremental synchronization. |
| Incremental CDC Type | Select a Change Data Capture (CDC) type based on the log engine of the PolarDB-X 2.0 instance. Multi-stream CDC requires multi-stream enabled on the source instance. |
| Processing Mode of Conflicting Tables | Precheck and Report Errors (default): DTS checks whether the destination database contains tables with the same names as those in the source. If duplicate names exist, the precheck fails and the task cannot start. To handle duplicate names without blocking the task, use the object name mapping feature to rename the destination tables. See Map object names. Ignore Errors and Proceed: skips the duplicate name check. During full data synchronization, existing records in the destination are kept if they share a primary key or unique key value with a source record. During incremental data synchronization, source records overwrite matching destination records. If the source and destination have different schemas, initialization may fail or only some columns may be synchronized. |
| Source Objects | Select one or more objects and click the right-arrow icon to add them to Selected Objects. Select databases or tables as the objects to synchronize. |
| Selected Objects | To rename a sync object or map it to a different destination object, right-click it and configure object name mapping. See Map object names. To remove an object, click it and then click the remove icon. To filter rows, right-click a table and specify WHERE conditions. See Specify filter conditions. To select which SQL operations to sync incrementally, right-click an object and select the operations. |
Renaming a sync object may cause other objects that depend on it to fail synchronization.
Step 4: Configure advanced settings
Click Next: Advanced Settings and configure the following parameters.
| Parameter | Description |
|---|---|
| Dedicated Cluster for Task Scheduling | By default, DTS schedules tasks to a shared cluster. For higher stability, purchase a dedicated cluster. See What is a DTS dedicated cluster. |
| Retry Time for Failed Connections | How long DTS retries a failed connection before the task fails. Valid values: 10–1440 minutes. Default: 720. We recommend setting this to a value greater than 30. If DTS reconnects within the specified window, the task resumes. Otherwise, the task fails. If multiple tasks share the same source or destination database, the shortest retry window applies to all of them. |
| Retry Time for Other Issues | How long DTS retries failed DDL or DML operations before the task fails. Valid values: 1–1440 minutes. Default: 10. We recommend setting this to a value greater than 10. This value must be smaller than the Retry Time for Failed Connections value. |
| Enable Throttling for Full Data Synchronization | Limits resource usage during full data synchronization. Configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s) to reduce load on the destination database. Available only when Full Data Synchronization is selected. |
| Enable Throttling for Incremental Data Synchronization | Limits resource usage during incremental data synchronization. Configure RPS of Incremental Data Synchronization and Data synchronization speed for incremental synchronization (MB/s). |
| Whether to delete SQL operations on heartbeat tables of forward and reverse tasks | Controls whether DTS writes heartbeat SQL operations to the source database. Yes: DTS does not write heartbeat operations. Latency information may appear elevated. No: DTS writes heartbeat operations. This may affect features such as physical backup and database cloning. |
| Environment Tag | (Optional) Assign a tag to identify the instance environment. |
| Monitoring and Alerting | No: alerting disabled. Yes: enables alerting. Specify an alert threshold and notification settings. See Configure monitoring and alerting when you create a DTS task. |
When DTS retries a connection, you are charged for the synchronization instance during the retry period. Release the instance promptly if the source and destination instances are no longer needed.
Step 5: Configure data verification
Click Next Step: Data Verification to set up data verification. See Configure a data verification task.
Step 6: Run the precheck and save the task
Click Next: Save Task Settings and Precheck.
To preview the API parameters for configuring this task programmatically, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.
DTS runs a precheck before the task can start. If any item fails:
Click View Details next to the failed item.
Fix the issue based on the check results.
Click Precheck Again.
If an alert is triggered:
For an alert that cannot be ignored, click View Details, fix the issue, and run the precheck again.
For an alert that can be ignored, click Confirm Alert Details, click Ignore in the dialog box, click OK, and then click Precheck Again.
Ignoring an alert item may cause data inconsistency and expose your business to potential risks.
Step 7: Purchase the synchronization instance
Wait until Success Rate reaches 100%, then click Next: Purchase Instance.
On the purchase page, configure the following parameters.
| Parameter | Description |
|---|---|
| Billing Method | Subscription: pay upfront for a set period. More cost-effective for long-term use. Pay-as-you-go: billed hourly. Suitable for short-term use. Release the instance when it is no longer needed to stop charges. |
| Resource Group Settings | The resource group for the synchronization instance. Default: default resource group. See What is Resource Management? |
| Instance Class | Select an instance class based on your required synchronization speed. See Instance classes of data synchronization instances. |
| Subscription Duration | Available when Subscription is selected. Options: 1–9 months, 1 year, 2 years, 3 years, or 5 years. |
Read and select Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start, then click OK in the dialog box.
The task appears in the task list. Monitor its progress from there.
What's next
After the synchronization task starts:
Monitor task status: Check the task list regularly for latency spikes or errors.
Verify data consistency: After full initialization completes, verify that the destination data matches the source. See Configure a data verification task.
Avoid writing to the destination from other sources: External writes to the destination database during synchronization can cause data inconsistency or task failure.