How to resolve high latency in DTS incremental tasks - Data Transmission Service

Latency above 1,000 milliseconds during incremental data migration or synchronization usually points to a resource bottleneck on the DTS instance, the source database, or the destination database. Work through the following causes in order. Each section describes how to confirm the cause and what to do about it.

Quick check first: If the latency appeared suddenly and nothing else changed, check Cause 11 first — the DTS console display can lag behind actual replication state.

Cause 1: DTS instance RPS limit reached

Data migration instances and data synchronization instances each have a records per second (RPS) limit. If write volume from the source database hits that limit, latency builds up. Large transactions are a common trigger.

How to confirm: In the DTS console, check the data volume involved in the task, or click Quick Diagnostics to see whether the current instance's RPS limit is reached. See Monitor task performance, Specifications of data synchronization instances, and Specifications of data migration instances.

Fix: Upgrade the data migration instance or data synchronization instance. See Upgrade a DTS instance.

Cause 2: Destination database write bottleneck

If the destination database instance is under-provisioned, its write performance becomes the bottleneck.

How to confirm: For an ApsaraDB RDS for MySQL destination, check CPU utilization, memory usage, and I/O load on the Monitoring and Alerts page in the ApsaraDB RDS console. See View monitoring information and Instance types for standard primary ApsaraDB RDS for MySQL instances (original x86 architecture).

Fix: Upgrade the destination database instance. For ApsaraDB RDS for MySQL, see Change instance specifications.

Cause 3: Source database read bottleneck or bandwidth exhausted

If the source database cannot supply data fast enough — due to IOPS limits or saturated network bandwidth — DTS falls behind.

How to confirm: For an ApsaraDB RDS for MySQL source, check IOPS and other metrics on the Monitoring and Alerts page in the ApsaraDB RDS console. See View monitoring information and Instance types for standard primary ApsaraDB RDS for MySQL instances (original x86 architecture).

Fix: Upgrade the source database instance specifications or increase its network bandwidth. For ApsaraDB RDS for MySQL, see Change instance specifications.

Cause 4: Hotspot updates

Tables without primary keys, or frequent updates concentrated on a single table or row, create hotspot updates that slow replication.

How to confirm: For an ApsaraDB RDS for MySQL source, check the SQL Explorer and Audit page in the ApsaraDB RDS console to see whether transactions are concentrated on specific tables. See Use the SQL Explorer and Audit feature.

Fix: Wait for the hotspot updates to finish. We recommend that you do not perform hotspot updates for business considerations.

Cause 5: DTS retrying due to destination write failure

If DTS cannot connect to the destination database or encounters a write exception, it retries repeatedly. Each retry cycle adds latency.

How to confirm: Check whether the task status is Retrying in the DTS console, then click Troubleshoot to view the specific error.

Example scenario: If the destination is an AnalyticDB for MySQL instance and DDL operations change field types on the source, the destination cannot apply those changes. DTS retries repeatedly without success, causing latency to grow.

Fix for the DDL example:

Remove the tables affected by DDL operations from the objects to be synchronized.
Wait until the synchronization latency drops to 0 milliseconds.
Remove those tables from the destination database.
Add the tables back to the objects to be synchronized.

See Data synchronization FAQ and Modify the objects to be synchronized.

Cause 6: Triggers in the destination database

Triggers on the destination database execute additional logic for every row written by DTS, increasing write latency.

How to confirm: Run the following query to check for triggers in a MySQL destination database:

SELECT * FROM information_schema.triggers WHERE trigger_schema='<Database name>';

Fix: Delete or disable the triggers in the destination database. To delete a trigger from a MySQL database:

DROP TRIGGER [IF EXISTS] [Database name].<Trigger name>;

If the source database contains triggers and you need to keep them, see Configure a data synchronization task for a source database that contains a trigger.

Cause 7: Complex table schemas causing locks in the destination database

If destination tables have both primary keys and unique keys, and other systems are also writing to those tables concurrently with DTS, table locks and slow SQL queries can occur.

How to confirm and fix (run these commands against the destination database):

Check locks and slow queries: Run SHOW PROCESSLIST; to identify locked tables or slow processes, then run KILL [CONNECTION | QUERY] <thread_id> to terminate them.
Check table schemas: Run SHOW CREATE TABLE <Database name>.<Table name>; to inspect the schema. If a unique key is not required, drop it:
```
ALTER TABLE <Database name>.<Table name> DROP INDEX <Unique key name>;
```

Cause 8: Excessive DDL operations on the source database

Running many DDL operations while a DTS task is active can cause latency because DTS must replicate each DDL statement to the destination.

How to confirm: For an ApsaraDB RDS for MySQL source, check the SQL Explorer and Audit page in the ApsaraDB RDS console for DDL activity. See Use the SQL Explorer and Audit feature.

Fix: Avoid running multiple DDL operations on the source database while a DTS task is running. If DDL operations are necessary, perform them during off-peak hours.

When configuring the task, select only the DDL operation types that the task actually needs to synchronize.

Cause 9: Network latency from long-distance transmission

Cross-region replication introduces network latency that cannot be eliminated.

How to confirm: Check the regions of the source and destination instances on the Data Migration and Data Synchronization pages in the DTS console.

Fix: Connect the source and destination instances over Express Connect to reduce cross-region network latency.

Cause 10: Schema differences between source and destination tables

If the source and destination tables have different schemas, DTS may need to transform data during replication, which adds overhead.

Fix: Align the table schemas in the source and destination databases based on your business requirements.

Cause 11: Display latency in the DTS console

The latency value shown in the DTS console has a short update interval. A high displayed value does not always indicate actual data replication lag.

Fix: Wait a few minutes and refresh the page to get an updated reading.

References

What do I do if full data tasks stop or slow down?