Data Transmission Service:Track data changes from a DMS logical database - Data Transmission Service

Data Transmission Service (DTS) can capture real-time incremental data changes from a Data Management (DMS) logical database. This is a change data capture (CDC) task — it tracks changes to an existing database and does not perform an initial full data load. Use this guide to create a change tracking task for a DMS logical database backed by sharded PolarDB for MySQL clusters.

How DTS tracks changes

DTS reads binary logs from each PolarDB for MySQL physical database shard that backs the DMS logical database, then streams incremental change events to your consumer. Because DTS relies on binary logs, all binary logging requirements in Limitations must be met before you start.

Prerequisites

Before you begin, ensure that you have:

A configured DMS logical database. See Logical database.
Binary logging enabled on each PolarDB for MySQL cluster, with binlog_row_image set to full.
Binary logs retained for more than 24 hours on each cluster. Shorter retention causes DTS to fail to read logs, which may result in task failure or data loss.
Source tables with PRIMARY KEY or UNIQUE constraints, and all fields unique. Tables without these constraints may produce duplicate change records.

Limitations

Source database

Constraint	Detail
Table constraints	Source tables must have `PRIMARY KEY` or `UNIQUE` constraints with all fields unique. Without these, tracked changes may contain duplicates.
Table count cap	When tracking by table (not by database), a single change tracking task supports up to 500 tables. Exceeding this limit causes a request error. To track more tables, create multiple tasks in batches, or track the entire database instead.
Binary logging	Binary logging must be enabled. Set `binlog_row_image` to `full`. Retain binary logs for more than 24 hours — shorter retention may cause DTS to miss log segments, resulting in task failure or data inconsistency. Failure to meet these requirements is not covered by the DTS Service Level Agreement (SLA).

Other

Constraint	Detail
FLOAT and DOUBLE precision	DTS retrieves values from `FLOAT` and `DOUBLE` columns using `ROUND(COLUMN,PRECISION)`. Default precision is 38 digits for `FLOAT` and 308 digits for `DOUBLE`. Verify that the default precision meets your requirements before creating the task.
DDL tools	DTS does not track DDL operations performed by gh-ost or pt-online-schema-change. Schema changes made with these tools may cause the change tracking client to fail when writing to destination tables.
Classic network	If the ApsaraDB RDS for MySQL instance uses a classic network, an internal endpoint is configured for it.

Create a change tracking task

Step 1: Open the Change Tracking Tasks page

Log on to the Data Management (DMS) console.
In the top navigation bar, click DTS.
In the left-side navigation pane, choose DTS (DTS) > Change Tracking.

If you use DMS in Simple Mode, move the pointer over the icon in the upper-left corner, then choose All functions > DTS > Change Tracking. Alternatively, open the new DTS console directly.

Step 2: Select a region

To the right of Change Tracking Tasks, select the region where you want to create the task.

In the new DTS console, select the region from the drop-down list to the right of Workbench on the Change Tracking Tasks page.

Step 3: Configure the source database and consumer network

Click Create Task.
Warning
After specifying the source database, read the Limits section at the top of the page before proceeding. Skipping this step may cause the task to fail or prevent tracked data from being consumed.

Configure the following parameters:

Task name

Parameter	Description
Task Name	A name for the change tracking task. DTS assigns a default name. Specify a descriptive name for easy identification — the name does not need to be unique.

Source database

Parameter	Description
Select an existing database connection	Select an existing connection template if available. DTS populates the remaining parameters automatically. To use a new connection, leave this blank and fill in the parameters below.
Database Type	Select DMS LogicDB.
Access Method	Select Alibaba Cloud Instance.
Instance Region	The region where the DMS logical database resides.
DMS LogicDB	The name of the DMS logical database.
Database Account	The username used to access the DMS logical database. All PolarDB for MySQL clusters hosting the physical databases must share the same username and password, and each cluster must have read permissions on its corresponding physical database. See Create and manage a database account.
Database Password	The password for the database account.

Consumer network type

Parameter	Description
Network Type	Fixed to VPC. Select a VPC and vSwitch. If the change tracking client is deployed in a VPC, select the same VPC and vSwitch as the client to minimize network latency. These settings cannot be changed after the task is created.

Step 4: Test connectivity

Click Test Connectivity and Proceed.

DTS automatically adds its server CIDR blocks to the whitelist or security group rules for Alibaba Cloud database instances and ECS-hosted databases. For self-managed databases in on-premises data centers or third-party clouds, manually add the DTS server CIDR blocks to the database security settings. See Add the CIDR blocks of DTS servers.

Warning

Adding public DTS server CIDR blocks to a database whitelist or ECS security group exposes the database to security risks. Before proceeding, take preventive measures such as: strengthening username and password security, limiting exposed ports, authenticating API calls, regularly auditing whitelist and security group rules, and connecting to DTS over Express Connect, VPN Gateway, or Smart Access Gateway.

Step 5: Configure tracked objects and advanced settings

Basic settings

Parameter	Description
Data Change Types	Select one or both: Data Update tracks INSERT, DELETE, and UPDATE operations on selected objects. Schema Update tracks create, delete, and modify operations on all object schemas in the source instance — use the change tracking client to filter events downstream.
Source Objects	Select objects in the Source Objects panel and click to move them to Selected Objects. Select a database to track all current and future objects in it. Select individual tables to track only those tables — adding more tables later requires modifying the task. See Modify the objects for change tracking.

Advanced settings

Parameter	Description
Monitoring and Alerting	Enable alerting to receive notifications when the task fails or latency exceeds a threshold. If you select Yes, configure the alert threshold and notification settings. See Configure monitoring and alerting when you create a DTS task.
Retry Time for Failed Connections	How long DTS retries a failed connection before marking the task as failed. Valid values: 10–1440 minutes. Default: 720 minutes. Set this to at least 30 minutes. If multiple change tracking tasks share the same source instance, the shortest retry time among all tasks takes precedence. DTS charges fees during retry periods — release the DTS instance promptly if the source database instance is released.
Configure ETL	Select Yes to configure extract, transform, and load (ETL) processing with data transformation statements. Select No to skip ETL. See What is ETL?

Step 6: Save settings and run precheck

Click Next: Save Task Settings and Precheck.

To preview the API parameters used to configure this task, move the pointer over the button and click Preview OpenAPI parameters.

DTS runs a precheck before starting the task. If any item fails:

Click View Details next to the failed item and fix the reported issue, then click Precheck Again.
If an item generates an alert that can be safely ignored, click Confirm Alert Details, then click Ignore in the dialog, confirm, and click Precheck Again. Ignoring alert items may result in data inconsistency.

Step 7: Wait for precheck to complete

Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

Step 8: Purchase the change tracking instance

On the Purchase page, configure the billing settings:

Parameter	Description
Billing method	Subscription: billed upfront for a fixed term; more cost-effective for long-term use, with lower prices for longer durations. Pay-as-you-go: billed hourly; suited for short-term use. Release a pay-as-you-go instance when it is no longer needed to stop charges.
Subscription Duration	Available for the Subscription billing method only. Choose 1–9 months, or 1, 2, 3, or 5 years.
Resource Group Settings	The resource group for this instance. Default: default resource group. See What is Resource Management?

Step 9: Accept terms and start the task

Read and select the Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start.

The change tracking task starts and appears in the task list.

Troubleshooting

Symptom	Likely cause	Action
Task fails with a log fetch error	Binary logs on the source cluster were purged before DTS could read them. This happens when the log retention period is less than 24 hours.	Increase the binary log retention period to more than 24 hours, then restart the task.
Task fails precheck on table constraints	One or more source tables lack `PRIMARY KEY` or `UNIQUE` constraints.	Add primary key or unique constraints to the affected tables, then run the precheck again.
Request error when creating task with more than 500 tables	The single-task table limit is 500.	Split the tables into multiple change tracking tasks, or track the entire database instead of individual tables.
Change tracking client fails to write to destination tables	A DDL operation was performed using gh-ost or pt-online-schema-change, which DTS does not track.	Avoid using gh-ost or pt-online-schema-change for schema changes on tracked tables, or manually apply the DDL change to the destination.

Next steps

After the task is running, create consumer groups so downstream clients can consume the tracked data:

Create and manage consumer groups. See Create consumer groups.
Choose a consumption method: