Use Data Transmission Service (DTS) to synchronize data from a PolarDB for MySQL cluster to the wide table engine of a Lindorm instance. DTS performs an initial full data load, then continuously applies incremental changes to keep the destination in sync.
Prerequisites
Before you begin, make sure you have:
A Lindorm instance using the LindormTable engine, with available storage space exceeding the storage space used by the source PolarDB for MySQL cluster. See Create an instance.
The MySQL compatibility feature enabled on the destination Lindorm instance. See Enable the MySQL compatibility feature.
A database (namespace) and a pre-partitioned wide table created in the destination Lindorm instance to receive the synchronized data. See Use a MySQL client to connect to and use LindormTable, Use Lindorm-cli to connect to and use the wide table engine, Use Lindorm Shell to access LindormTable, CREATE TABLE, and Data type mapping.
Binary logging enabled on the source PolarDB for MySQL cluster, with the
loose_polar_log_binparameter set toON. See Enable binary logging and Modify parameters.Enabling the binary logging feature for a PolarDB for MySQL cluster incurs additional charges for the storage space occupied by binary logs.
Binary log retention set to at least three days on the source cluster (seven days recommended). See Modify the retention period. Insufficient retention can cause data inconsistency or loss and may affect the DTS Service-Level Agreement (SLA).
Database accounts with the required permissions. See Permissions required for database accounts.
Objects created in the destination Lindorm instance must meet the requirements specified in Quotas and limits.
Billing
| Synchronization type | Fee |
|---|---|
| Full data synchronization | Free of charge |
| Incremental data synchronization | Charged. See Billing overview. |
SQL operations supported for incremental synchronization
| Operation type | SQL statements |
|---|---|
| DML | INSERT, UPDATE, DELETE |
| DDL | CREATE TABLE, DROP TABLE, ADD COLUMN |
Due to LindormTable engine limitations, DTS strips extra attributes from ADD COLUMN statements. For example, if the source runsALTER TABLE test ADD COLUMN col INT DEFAULT 0;, DTS runsALTER TABLE test ADD COLUMN col INT;on the destination.
Permissions required for database accounts
| Database | Required permissions | Reference |
|---|---|---|
| Source PolarDB for MySQL cluster | Read and write permissions | Create and manage a database account |
| Destination Lindorm instance | Read and write permissions on the destination namespace | Manage users |
Limitations
Schema synchronization is not supported. Only full data synchronization and incremental data synchronization are available.
Data of the BIT type cannot be synchronized.
Data to be synchronized must contain at least one non-primary key field. Synchronizing only primary key fields is not supported.
Tables to be synchronized must have PRIMARY KEY or UNIQUE constraints with all fields unique. Otherwise, the destination may contain duplicate records.
When selecting tables as synchronization objects and renaming tables or columns in the destination, a single task supports a maximum of 1,000 tables. Exceeding this limit causes a request error. In this case, configure multiple tasks or synchronize the entire database instead.
DTS does not synchronize read-only nodes of the source PolarDB for MySQL cluster.
DTS does not synchronize Object Storage Service (OSS) external tables from the source PolarDB for MySQL cluster.
DTS does not support primary/secondary switchover during full data synchronization. If a switchover occurs, reconfigure the synchronization task.
Data can only be written to the wide table engine of the Lindorm instance.
Empty strings of the VARBINARY type are treated as null values in DTS and the Lindorm instance.
If the DECIMAL type precision differs between source and destination fields, the task fails.
Data in the Lindorm instance must meet the requirements in Limits on data requests. Otherwise, the task fails.
Do not perform DDL operations on the source database during full data synchronization — this causes the task to fail. DTS queries the source during this phase, which creates metadata locks that can also block DDL operations on the source.
Do not use tools such as pt-online-schema-change to perform online DDL operations on synchronized objects during synchronization. This causes the task to fail.
If non-DTS data is written to the destination during synchronization, data inconsistency may occur.
During initial full data synchronization, concurrent INSERT operations cause table fragmentation in the destination. After the initial full data synchronization completes, the destination table storage space will be larger than the source.
During initial full data synchronization, DTS uses read and write resources of both databases, which may increase server load. Schedule synchronization during off-peak hours — for example, when the CPU load of both databases is below 30%.
DTS periodically runs
CREATE DATABASE IF NOT EXISTS \test\`` on the source database to advance the binary log file position.If a DTS instance fails, the DTS team will attempt recovery within 8 hours. Recovery may involve restarting the instance or adjusting DTS instance parameters (database parameters are not modified). Parameters that may be adjusted include those described in Modify instance parameters.
Configure a synchronization task
Step 1: Open the Data Synchronization page
Use one of the following methods:
DTS console
Log on to the DTS console
In the left-side navigation pane, click Data Synchronization.
In the upper-left corner, select the region where the synchronization task resides.
DMS console
The actual steps may vary based on the mode and layout of the DMS console. See Simple mode and Customize the layout and style of the DMS console.
Log on to the DMS console
In the top navigation bar, move the pointer over Data + AI and choose DTS (DTS) > Data Synchronization.
From the drop-down list to the right of Data Synchronization Tasks, select the region where the synchronization instance resides.
Step 2: Configure source and destination databases
Click Create Task.
Configure the task parameters as described in the following table.
Section Parameter Description N/A Task Name Enter a descriptive name to identify the task. A unique name is not required; DTS generates a default name automatically. Source Database Select Existing Connection If the source instance is already registered with DTS, select it from the drop-down list — DTS fills in the parameters automatically. Otherwise, configure the parameters manually. In the DMS console, select from the Select a DMS database instance drop-down list. Database Type Select PolarDB for MySQL. Access Method Select Alibaba Cloud Instance. Instance Region Select the region where the source PolarDB for MySQL cluster resides. Replicate Data Across Alibaba Cloud Accounts Select No to synchronize within the same Alibaba Cloud account. PolarDB Cluster ID Select the ID of the source PolarDB for MySQL cluster. Database Account Enter the database account for the source cluster. See Permissions required for database accounts. Database Password Enter the password for the database account. Encryption Select an SSL encryption option as needed. See Set SSL encryption. Destination Database Select Existing Connection If the destination instance is already registered with DTS, select it from the drop-down list. Otherwise, configure the parameters manually. In the DMS console, select from the Select a DMS database instance drop-down list. Database Type Select Lindorm. Access Method Select Alibaba Cloud Instance. Instance Region Select the region where the destination Lindorm instance resides. Instance ID Select the ID of the destination Lindorm instance. Database Account Enter the database account for the destination Lindorm instance. See Permissions required for database accounts. Database Password Enter the password for the database account. Click Test Connectivity and Proceed.
Make sure the CIDR blocks of DTS servers are added to the security settings of both databases to allow access. See Add DTS server IP addresses to a whitelist. If the source or destination is a self-managed database with an Access Method other than Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.
Step 3: Select objects to synchronize
In the Configure Objects step, configure the following parameters.
| Parameter | Description |
|---|---|
| Synchronization Types | Incremental Data Synchronization is selected by default. Optionally add Full Data Synchronization. Schema Synchronization is not available for this source-destination pair. |
| Processing Mode of Conflicting Tables | Use the default value. |
| Capitalization of Object Names in Destination Instance | Controls the capitalization of database names, table names, and column names in the destination. DTS default policy is selected by default. See Specify the capitalization of object names in the destination instance. |
| Source Objects | Select objects from Source Objects and click the  icon to move them to Selected Objects. You can select columns, tables, and databases. |
| Selected Objects | If the names of databases (namespaces), tables, or columns in the destination differ from those in the source, use the object name mapping feature. See Map object names. To filter rows with WHERE conditions, right-click a table in Selected Objects and specify the conditions. See Specify filter conditions. To select specific SQL operations for incremental synchronization, right-click an object in Selected Objects and choose the operations. |
Renaming objects with the object name mapping feature may cause dependent objects to fail synchronization.
Step 4: Configure advanced settings
Click Next: Advanced Settings and configure the following parameters.
| Parameter | Description |
|---|---|
| Dedicated Cluster for Task Scheduling | DTS uses the shared cluster by default. For higher stability, purchase a dedicated cluster. See What is a DTS dedicated cluster. |
| Retry Time for Failed Connections | How long DTS retries after a connection failure. Valid values: 10–1,440 minutes. Default: 720 minutes. Set to more than 30 minutes. If DTS reconnects within this window, the task resumes; otherwise, the task fails. If multiple tasks share the same source or destination, the shortest retry time applies. During retry, the DTS instance continues to incur charges. |
| Retry Time for Other Issues | How long DTS retries after DDL or DML failures. Valid values: 1–1,440 minutes. Default: 10 minutes. Set to more than 10 minutes. Must be less than Retry Time for Failed Connections. |
| Enable Throttling for Full Data Synchronization | Limits the load DTS places on source and destination during full data synchronization. Configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s) as needed. Available only when Full Data Synchronization is selected. |
| Enable Throttling for Incremental Data Synchronization | Limits the load during incremental data synchronization. Configure RPS of Incremental Data Synchronization and Data synchronization speed for incremental synchronization (MB/s) as needed. |
| Whether to delete SQL operations on heartbeat tables of forward and reverse tasks | Controls whether DTS writes heartbeat table operations to the source database. Yesalert notification settings: does not write heartbeat operations (a latency value may appear on the task). No: writes heartbeat operations (may affect physical backup and cloning of the source database). |
| Environment Tag | Optionally select a tag to identify the environment. |
| Configure ETL | Specifies whether to enable the extract, transform, and load (ETL) feature. Yes: enter data processing statements in the code editor. See Configure ETL in a data migration or data synchronization task. No: skip ETL configuration. See What is ETL? |
| Monitoring and Alerting | Configures alerts when the task fails or synchronization latency exceeds a threshold. Yes: configure the alert threshold and notification contacts. See Configure monitoring and alerting when you create a DTS task. No: no alerts. |
Step 5: Run a precheck
Click Next: Save Task Settings and Precheck.
To preview the API parameters for this task configuration, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters before proceeding.
DTS runs a precheck before the task can start. If the precheck fails:
Click View Details next to each failed item, resolve the issues, and click Precheck Again.
If an alert item can be ignored, click Confirm Alert Details, then Ignore, and click Precheck Again. Ignoring alerts may result in data inconsistency.
Step 6: Purchase an instance
Wait until Success Rate reaches 100%, then click Next: Purchase Instance.
On the buy page, configure the instance parameters.
Parameter Description Billing Method Subscription: pay upfront for a fixed period. More cost-effective for long-term use. Pay-as-you-go: billed hourly. Suitable for short-term use; release the instance when no longer needed to stop charges. Resource Group Settings The resource group for the synchronization instance. Default: default resource group. See What is Resource Management? Instance Class The synchronization speed varies by instance class. Select a class based on your throughput requirements. See Instance classes of data synchronization instances. Subscription Duration Available for the Subscription billing method only. Options: 1–9 months, or 1, 2, 3, or 5 years. Read and select Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start, then click OK in the dialog box.
The task appears in the task list. Monitor its progress there.