Data Transmission Service (DTS) enables real-time data synchronization from a PolarDB-X 2.0 instance to a Tablestore instance. DTS supports schema synchronization, full data synchronization, and incremental data synchronization (INSERT, UPDATE, DELETE), giving you a continuously updated copy of your source data in Tablestore.
Prerequisites
Before you begin, make sure that you have:
A Tablestore instance. For instructions, see Activate Tablestore and create an instance.
The AccessKey ID and AccessKey secret of the Alibaba Cloud account that owns the Tablestore instance. For instructions, see Create an AccessKey pair for an Alibaba Cloud account.
Billing
| Synchronization type | Pricing |
|---|---|
| Schema synchronization and full data synchronization | Free |
| Incremental data synchronization | Charged. For details, see Billing overview. |
SQL operations that can be incrementally synchronized
| Operation type | SQL statement |
|---|---|
| DML | INSERT, UPDATE, DELETE |
Limitations
Review these limitations before creating a synchronization task.
Source database limits
Tables must have a primary key or a unique constraint with unique fields. Otherwise, the destination database may contain duplicate records.
Read-only instances of PolarDB-X 2.0 Enterprise Edition are not supported as the source.
If you synchronize at the table level with object editing (such as column name mapping) and a single task includes more than 5,000 tables, split the task or configure it to synchronize the entire database instead.
If a source table name contains uppercase letters, only schema synchronization is supported for that table.
Binlog is enabled by default in PolarDB-X 2.0. Make sure the
binlog_row_imageparameter is set tofull. For instructions, see Parameter settings. If this parameter is not set correctly, the precheck fails and the task cannot start.Table groups and databases or tables with the Locality property are not supported.
Tables whose names are reserved words (such as
select) are not supported.Do not run DDL operations that change database or table schemas during schema synchronization or full data synchronization. This causes the synchronization task to fail.
NoteDuring full data synchronization, DTS queries the source database, which creates metadata locks that may block DDL operations on the source database.
DRDS-mode database partitions in PolarDB-X 2.0 cannot be synchronized.
Other limits
Evaluate the performance of your source and destination databases before starting synchronization. Run synchronization tasks during off-peak hours. Initial full data synchronization consumes read and write resources on both databases, which increases their load.
Initial full data synchronization runs concurrent INSERT operations, which causes table fragmentation in Tablestore. As a result, the tablespace of the Tablestore instance will be larger than that of the source instance after initial full data synchronization completes.
Do not write data to the destination database from sources other than DTS during synchronization. This can cause data inconsistency between source and destination.
The number of tables to synchronize must not exceed the Tablestore instance limit of 64. To synchronize more than 64 tables, contact Tablestore support to increase this limit.
Table and column names must comply with Tablestore naming conventions:
Names can contain uppercase letters, lowercase letters, digits, and underscores (
_), and must start with a letter or an underscore.Names must be 1–255 characters in length.
If a task fails, DTS support staff will attempt to restore it within 8 hours. During restoration, they may restart the task or adjust DTS task parameters—not database parameters. For the parameters that may be adjusted, see Modify instance parameters.
Create a synchronization task
The full configuration process consists of the following steps:
Go to the Data Synchronization task list.
Configure the source and destination databases.
Configure synchronization objects and Tablestore write settings.
Configure advanced settings.
Configure primary key columns.
Run the precheck and purchase the instance.
Step 1: Go to the Data Synchronization task list
Open the task list using either the DTS console or the DMS console.
DTS console
Log on to the DTS console.DTS console
In the left navigation pane, click Data Synchronization.
In the upper-left corner, select the region where the synchronization instance will be located.
DMS console
The exact steps may vary depending on the mode and layout of the DMS console. For details, see Simple mode console and Customize the layout and style of the DMS console.
Log on to the DMS console.DMS console
In the top menu bar, choose Data + AI > DTS (DTS) > Data Synchronization.
To the right of Data Synchronization Tasks, select the region of the synchronization instance.
Step 2: Configure the source and destination databases
Click Create Task.
Enter a Task Name. DTS generates one automatically, but a descriptive name makes identification easier. The name does not need to be unique.
Configure the source database.
Parameter Description Select Existing Connection Select a registered database instance from the drop-down list to auto-fill the settings below. If you have not registered the instance or prefer to configure it manually, skip this field. NoteIn the DMS console, this field is labeled Select a DMS database instance.
Database Type Select PolarDB-X 2.0. Access Method Select Alibaba Cloud Instance. Instance Region Select the region where the source PolarDB-X 2.0 instance is located. Replicate Data Across Alibaba Cloud Accounts Select No (this example uses the same Alibaba Cloud account). Instance ID Select the ID of the source PolarDB-X 2.0 instance. Database Account Enter the database account for the source instance. The account must have the REPLICATION SLAVE, REPLICATION CLIENT, and SELECT permissions on the objects to synchronize. For instructions on granting permissions, see Manage database accounts and Account permissions required for data synchronization. Database Password Enter the password for the database account. Configure the destination database.
Parameter Description Select Existing Connection Select a registered database instance from the drop-down list to auto-fill the settings below. If you have not registered the instance or prefer to configure it manually, skip this field. NoteIn the DMS console, this field is labeled Select a DMS database instance.
Database Type Select Tablestore. Access Method Select Alibaba Cloud Instance. Instance Region Select the region where the destination Tablestore instance is located. Instance ID Select the name of the destination Tablestore instance. AccessKey ID of Alibaba Cloud Account Enter the AccessKey ID of the Alibaba Cloud account that owns the Tablestore instance. This is used for identity verification. ImportantIf you enter the AccessKey ID of a RAM user, the task may fail due to insufficient permissions.
AccessKey Secret of Alibaba Cloud Account Enter the AccessKey secret of the Alibaba Cloud account. Click Test Connectivity and Proceed at the bottom of the page.
NoteMake sure the DTS server IP address blocks are added to the security settings of your source and destination databases—either automatically or manually—before testing connectivity. For instructions, see Add the IP address whitelist of DTS servers.
Step 3: Configure synchronization objects and write settings
On the Configure Objects page, configure the following settings.
Synchronization scope
| Parameter | Description |
|---|---|
| Synchronization Types | DTS always enables Incremental Data Synchronization, which continuously applies ongoing changes from source to destination. By default, Schema Synchronization and Full Data Synchronization are also selected—DTS first copies the full data from the selected objects to initialize the destination, then switches to incremental synchronization. |
| Operation Types | Select the DML operation types to synchronize. INSERT, UPDATE, and DELETE are all selected by default. |
Conflict handling
| Parameter | Description |
|---|---|
| Processing Mode of Conflicting Tables | Controls how DTS handles name conflicts between source and destination tables. Precheck and Report Errors: DTS checks for name conflicts before the task starts. If conflicts are found, an error is reported and the task does not start. To resolve conflicts without renaming destination tables, you can map them to different names—see Database Table Column Name Mapping. Ignore Errors and Proceed: DTS skips the conflict check. Warning This may cause data inconsistency. During full data synchronization, DTS skips source records that conflict with existing destination records (same primary or unique key). During incremental synchronization, DTS overwrites destination records with source records. If table schemas are inconsistent, initialization may fail or result in partial synchronization. Use with caution. |
| Processing Policy of Dirty Data | Controls how DTS handles data write errors. Skip: Skips records that fail to write. Block: Stops the task when a write error occurs. |
Tablestore write settings
| Parameter | Description |
|---|---|
| Data Write Mode | Controls how DTS writes rows to Tablestore. Overwrite Row: Uses UpdateRowChange for row-level overwrites. Update Row: Uses PutRowChange for row-level updates. |
| Batch Write Mode | The API used for batch writes. BulkImportRequest (recommended): Offline write mode with higher read and write efficiency and lower Tablestore billing costs. BatchWriteRowRequest: Standard batch write mode. |
| More (optional) | Expand to configure advanced write performance parameters: Queue Size (write queue length), Thread Quantity (callback handler threads), Concurrency (concurrent request limit), and Buckets (concurrent buckets for sequential writing of incremental data—increasing this value improves concurrent write performance, but it must not exceed Concurrency). |
Object selection and mapping
| Parameter | Description |
|---|---|
| Capitalization of Object Names in Destination Instance | The case policy for database, table, and column names in the destination. The DTS default policy is selected by default. For details, see Case policy for destination object names. |
| Source Objects | In the Source Objects box, click the objects to synchronize, then click the arrow icon to move them to the Selected Objects box. You can select entire databases or individual tables. Selecting tables does not synchronize views, triggers, or stored procedures. You can synchronize tables from only a single database per task. |
| Selected Objects | To rename an object in the destination, right-click it in the Selected Objects box. For single or bulk renaming details, see Map object names. When using column name mapping, you can also change the data type of the corresponding column in the destination Tablestore instance. To filter rows, right-click a table and set a WHERE clause. For details, see Set filter conditions. Note Database name mapping is not supported. Table and column mapping is only available when you select tables (not entire databases) as the synchronization objects. |
Step 4: Configure advanced settings
Click Next: Advanced Settings and configure the following parameters.
| Parameter | Description |
|---|---|
| Dedicated Cluster for Task Scheduling | By default, DTS uses a shared cluster. To improve task stability, purchase a dedicated cluster. For details, see What is a DTS dedicated cluster? |
| Retry Time for Failed Connections | How long DTS retries after a connection to the source or destination fails. Default: 720 minutes. Range: 10–1,440 minutes. Set this to 30 minutes or more. If the connection is restored within this period, the task resumes automatically. Note If multiple DTS instances share a source or destination, DTS applies the shortest configured retry duration across all of them. DTS charges for task runtime during connection retries. |
| Retry Time for Other Issues | How long DTS retries after a non-connection error (such as a DDL or DML execution error). Default: 10 minutes. Range: 1–1,440 minutes. Set this to 10 minutes or more. Important This value must be less than Retry Time for Failed Connections. |
| Enable Throttling for Full Data Synchronization | Limit the full data synchronization rate to reduce load on your databases. Configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s). This parameter is only available when Full Data Synchronization is selected. You can also adjust the rate after the task starts. |
| Enable Throttling for Incremental Data Synchronization | Limit the incremental synchronization rate by configuring RPS of Incremental Data Synchronization and Data synchronization speed for incremental synchronization (MB/s). |
| Whether to delete SQL operations on heartbeat tables of forward and reverse tasks | Controls whether DTS writes heartbeat SQL to the source database. Yesalert notifications: Does not write heartbeat SQL. The DTS instance may show higher latency. No: Writes heartbeat SQL to the source database. This may interfere with operations like physical backups and database cloning. |
| Environment Tag | Optionally select a tag to identify this instance. |
| Configure ETL | ETL is not supported for this synchronization path. Select No. |
| Monitoring and Alerting | Configure alerts for task failures or latency exceeding a threshold. No: No alerts. Yes: Configure alert thresholds and recipients. For details, see Configure monitoring and alerting. |
Step 5: Configure primary key columns
Click Next: Configure Database and Table Fields, then click OK in the dialog box that appears.
DTS sets the primary key of each source table as the Primary Key Column by default. To modify this, set Definition Status to All.
You can select multiple columns to form a composite primary key.
Step 6: Run the precheck and purchase the instance
To preview the API parameters for this configuration, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters in the tooltip.
Click Next: Save Task Settings and Precheck. DTS runs a precheck before starting the task. The task can only start after the precheck passes.
If the precheck fails, click View Details next to the failed item, fix the issue as instructed, and rerun the precheck.
If the precheck generates warnings:
For non-ignorable warnings, click View Details, fix the issue, and rerun the precheck.
For ignorable warnings, click Confirm Alert Details > Ignore > OK, then click Precheck Again. Ignoring precheck warnings may cause data inconsistency. Proceed with caution.
When the Success Rate reaches 100%, click Next: Purchase Instance.
On the Purchase page, select the billing method and instance specifications.
Parameter Description Billing Method Subscription: Pay upfront for a fixed duration. Best for long-term, continuous synchronization tasks. Pay-as-you-go: Billed hourly for actual usage. Best for short-term or test tasks—release the instance at any time to stop billing. Resource Group Settings The resource group for this instance. Defaults to default resource group. For details, see What is Resource Management? Instance Class Different specifications offer different synchronization throughput levels. Select based on your workload requirements. For details, see Data synchronization link specifications. Subscription Duration Available only for Subscription billing. Monthly options: 1–9 months. Yearly options: 1, 2, 3, or 5 years. Read and select the checkbox for Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start, then click OK in the confirmation dialog.
Monitor the task progress on the Data Synchronization page.