Q: Which database instances does DTS support for synchronization?
A: DTS supports data synchronization between various data sources, such as relational database management systems (RDBMSs), NoSQL databases, and online analytical processing (OLAP) databases. For the supported database instances, see Data synchronization scenarios.
Q: How does data synchronization work?
A: For information about how data synchronization works, see Product architecture and how features work.
Q: How is synchronization latency calculated?
A: Synchronization latency is the difference between the timestamp of the latest data synchronized to the destination database and the current timestamp of the source database. The unit is milliseconds. Under normal conditions, the latency is within 1,000 milliseconds.
Q: Can I modify the objects to synchronize in a data synchronization task?
A: Yes. To learn how to modify the objects to synchronize, see Add a synchronization object and Remove a synchronization object.
Q: Can I add new tables to a data synchronization task?
A: Yes. To learn how to add new tables, see Add a synchronization object.
Q: How do I modify objects to synchronize, such as tables and columns, in a running synchronization task?
A: After the full data synchronization stage ends and the task enters the incremental data synchronization stage, you can modify the objects to synchronize. To learn how to modify the objects to synchronize, see Add a synchronization object and Remove a synchronization object.
Q: If I suspend a synchronization task and restart it after a while, will data inconsistency occur?
A: If the source database changes while the synchronization task is suspended, data inconsistency between the source and destination databases may occur. After you restart the task and the incremental data is synchronized to the destination database, the data in the destination database becomes consistent with that in the source database.
Q: If I delete data from the source database of an incremental synchronization task, will the synchronized data in the destination database also be deleted?
A: If delete is not selected for the DML operations to synchronize in the incremental synchronization task, the data in the destination database is not deleted. Otherwise, the synchronized data in the destination database is deleted.
Q: In Redis-to-Redis synchronization, is the data in the destination Redis instance overwritten?
A: Data with the same key is overwritten. DTS checks the destination during the precheck stage. If the destination is not empty, an error is reported.
Q: Does a synchronization task support filtering out specific columns or data?
A: Yes. You can use the mapping feature to filter out columns that do not need to be synchronized, and specify SQL WHERE conditions to filter the data to synchronize. For more information, see Synchronize or migrate specific columns and Filter task data with an SQL condition.
Q: Can I convert a synchronization task into a migration task?
A: No. Different types of tasks cannot be converted to each other.
Q: Can I synchronize only the data without synchronizing the schema?
A: Yes. When you configure the synchronization task, do not select Schema Synchronization.
Q: What are the possible causes of data inconsistency between the source and destination of a data synchronization instance?
The possible causes of data inconsistency are as follows:
When you configured the task, you did not clear the destination, and the destination contained existing data.
When you configured the task, you selected only the incremental synchronization module and did not select the full synchronization module.
When you configured the task, you selected only the full synchronization module and did not select the incremental synchronization module, and the source data changed after the task ended.
Data other than that written by the DTS task was written to the destination.
Incremental writes are delayed, and not all incremental data has been written to the destination.
Q: Can I modify the name of the source database in the destination database in a data synchronization task?
A: Yes. To learn how to modify the name of the source database in the destination database, see Rename synchronized objects in the destination instance.
Q: Does DTS support real-time synchronization of DML or DDL operations?
Yes. For data synchronization between relational databases, the supported DML operations are INSERT, UPDATE, and DELETE, and the supported DDL operations are CREATE, DROP, ALTER, RENAME, and TRUNCATE.
NoteThe supported DML and DDL operations vary based on the scenario. In Data synchronization scenarios, select the synchronization link that matches your business scenario, and view the supported DML and DDL operations in the configuration guide for the specific link.
Q: Can a read-only instance be used as the source instance of a synchronization task?
A synchronization task includes incremental data synchronization by default. Therefore, the following two scenarios apply:
If the instance is a read-only instance that records transaction logs (for example, ApsaraDB RDS for MySQL 5.7 or 8.0), it can serve as the source instance.
If the instance is a read-only instance that does not record transaction logs (for example, ApsaraDB RDS for MySQL 5.6), it cannot serve as the source instance.
Q: Does DTS support data synchronization for sharded databases and tables?
A: Yes. For example, you can synchronize sharded databases and tables in MySQL or PolarDB for MySQL to AnalyticDB for MySQL to merge multiple tables.
Q: Why is the data volume in the destination instance smaller than that in the source instance after the synchronization task ends?
A: If data was filtered during synchronization, or if the tables in the source instance contain many fragments, the data volume in the destination instance may be smaller than that in the source instance after the migration is complete.
Q: Do cross-account data synchronization tasks support two-way synchronization?
A: Currently, cross-account two-way synchronization tasks are supported only between ApsaraDB RDS for MySQL instances, between PolarDB for MySQL clusters, between Tair (Enterprise Edition) instances, between ApsaraDB for MongoDB (replica set architecture) instances, and between ApsaraDB for MongoDB (sharded cluster architecture) instances.
Q: Does DTS support cross-border two-way synchronization tasks?
A: No.
Q: Why doesn't a record added to one database in a two-way synchronization task appear in the other database?
A: The reverse task may not have been configured.
Q: Why does the progress of an incremental synchronization task never reach 100%?
A: DTS incremental synchronization continuously replicates changes from the source to the destination in real time and does not end on its own. Therefore, there is no 100% completion state. If you no longer need real-time synchronization, end the task in the DTS console.
Q: Why does the incremental synchronization task fail to synchronize data?
A: If only incremental synchronization is configured for the DTS instance, DTS synchronizes only the incremental data generated after the task starts. Data that exists before the task starts is not synchronized to the destination database. To ensure data consistency, we recommend that you select Incremental Data Synchronization, Schema Synchronization, and Full Data Synchronization when you configure the task.
Q: Does synchronizing full data from an RDS database affect the performance of the source RDS database?
Yes. The query performance of the source database is affected. You can use one of the following three methods to reduce the impact of the DTS task on the source database:
Upgrade the specifications of the source database instance.
Pause the DTS task and restart it when the load on the source database is low.
Reduce the rate of the DTS task. Adjust the full migration rate.
Q: Does DTS support synchronization from MySQL to ClickHouse?
A: The feature of synchronizing data from a MySQL database to a ClickHouse cluster in DTS has been discontinued. Users who have already purchased this feature are not affected. To newly purchase this feature, refer to the ApsaraDB for ClickHouse product documentation and see the Synchronize data from ApsaraDB RDS for MySQL topic to configure synchronization from MySQL to ClickHouse.
Q: Why is no latency displayed for a synchronization instance whose source database is PolarDB-X 1.0?
A: An instance whose source database is PolarDB-X 1.0 is a distributed task, but the metrics that DTS monitors exist only in subtasks. Therefore, no latency information is displayed for instances whose source is PolarDB-X 1.0. To view latency information, click the instance ID and go to Task Management > Subtask Details for the instance.
Q: Why does a multi-table merging task report the DTS-071001 error?
A: This may be because an online DDL operation was performed on the source database while the multi-table merging task was running, which modified the source table schema, and the corresponding changes were not manually applied to the destination database.
Q: What do I do if adding a whitelist fails when I configure a task in the previous console?
A: Configure the task in the new console.
Q: What do I do if a DTS data synchronization task fails because a DDL operation was performed on the source database?
A: Manually execute the DDL operation on the destination based on the DDL operation performed on the source, and then restart the task. During data synchronization, do not use tools such as pt-online-schema-change to perform online DDL changes on the objects to synchronize in the source database. Otherwise, synchronization fails. If no data other than that written by DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL changes, or you can modify the objects to synchronize to remove the table affected by the DDL. For the procedure to remove the table, see Remove a synchronization object.
Q: What do I do if a DTS data synchronization task fails because a DDL operation was performed on the destination database?
If a database or table in the destination is dropped during incremental synchronization by DTS and the task becomes abnormal, you can use one of the following two solutions to recover the task:
Solution 1: Reconfigure the task and do not select the database or table that caused the task to fail as an object to synchronize.
Solution 2: Modify the objects to synchronize and remove the database or table that caused the task to fail. For the procedure, see Remove a synchronization object.
Q: Can a synchronization task be recovered after it is released? Can data consistency be ensured by reconfiguring the task?
A: A released synchronization task cannot be recovered. When you reconfigure the task, if you do not select Full Data Synchronization, the data added during the period from task release to the start of the new task cannot be synchronized to the destination database, and data consistency cannot be ensured. If your business requires strict data accuracy, you can delete the data from the destination database, then reconfigure the synchronization task, and select both Schema Synchronization and Full Data Synchronization in Task Stages (Incremental Data Synchronization is selected by default).
Q: What do I do if a DTS full synchronization task shows no progress for a long time?
A: If the tables to synchronize do not have primary keys, full synchronization is very slow. We recommend that you add primary keys to the tables in the source database before you synchronize them.
Q: When you synchronize tables with the same name, can data from the source table be transferred only if it does not exist in the destination table?
A: Yes. When you configure the task, set Processing Mode of Conflicting Tables to Ignore Errors and Proceed. If the schemas are consistent, during full synchronization, when the destination encounters a record whose primary key value is the same as that in the source, the record in the source is not synchronized to the destination.
Q: How do I configure a cross-account synchronization task?
A: You must first determine Scenarios for cross-account tasks, then use the Alibaba Cloud account that owns the database instance to Configure RAM authorization for cross-account DTS tasks, and then Configure a cross-account DTS task.
Q: What do I do if I cannot select a DMS LogicDB instance?
A: Make sure that you selected the correct region to which the instance belongs. If you still cannot select the instance, only one instance may be available. Continue to configure the other parameters.
Q: Does a synchronization task that uses SQL Server as the source support synchronizing functions?
A: No. If tables are selected as the granularity of the objects to synchronize, other objects such as views, triggers, and stored procedures are not synchronized to the destination database.
Q: What do I do if a data synchronization task reports an error?
A: Based on the error message, view the solution in Overview of Common Errors.
Q: How do I enable hotspot merging for a synchronization task?
A: Refer to Modify parameter values to change the value of trans.hot.merge.enable to true.
Q: How do I perform synchronization when the source database contains triggers?
A: When an entire database is selected as an object to synchronize and a TRIGGER in the database updates a table in the database, data inconsistency between the source and destination databases may occur. For the synchronization procedure, see Configure tasks for source databases with triggers.
Q: Does DTS support synchronizing the sys database and system databases?
A: No.
Q: Does DTS support synchronizing the admin and local databases of MongoDB?
A: No. DTS does not support the admin and local databases of MongoDB as the source or destination database.
Q: When can the reverse task of a two-way synchronization task be configured?
A: The reverse task of a two-way synchronization task can be configured only after the forward incremental task has no latency.
Q: When PolarDB-X 1.0 is the source of a synchronization task, does the source PolarDB-X 1.0 support scaling nodes?
A: No. If the source PolarDB-X 1.0 scales nodes up or down, you must reconfigure the task.
Q: Can DTS ensure the uniqueness of data synchronized to Kafka?
A: No. Because data is appended to Kafka, duplicate data may occur when the DTS task is restarted or when the source logs are pulled multiple times. DTS ensures data idempotence: data is arranged in order, and the latest value of duplicate data is placed at the end.
Q: Does DTS data synchronization support synchronization from ApsaraDB RDS for MySQL to other destinations?
A: Yes. For the configuration procedure, see Synchronize data from ApsaraDB RDS for MySQL to AnalyticDB for MySQL 3.0.
Q: Why is full synchronization not displayed for a synchronization task between Redis instances?
A: Synchronization between Redis instances supports both full data synchronization and incremental data synchronization. The two stages are combined and displayed as Incremental Data Synchronization.
Q: Can I skip full synchronization?
A: Yes. After you skip full synchronization, incremental synchronization continues, but errors may occur. We recommend that you do not skip full synchronization.
Q: Does DTS support scheduled automatic synchronization?
A: DTS does not support scheduled starts of data synchronization tasks at this time.
Q: Is the fragmented space in tables also synchronized during synchronization?
A: No.
Q: What do I need to know when I synchronize data from MySQL 8.0 to MySQL 5.6?
A: You must create the databases in MySQL 5.6 before you perform synchronization. We recommend that the source and destination databases run the same version, or that you synchronize from a lower version to a higher version to ensure compatibility. Synchronizing from a higher version to a lower version may cause database compatibility issues.
Q: Can I synchronize accounts from the source database to the destination database?
A: Currently, only synchronization tasks between ApsaraDB RDS for MySQL instances support account synchronization. Other synchronization tasks do not support this feature.
Q: Can I configure cross-account two-way synchronization tasks?
Currently, cross-account two-way synchronization tasks are supported only between ApsaraDB RDS for MySQL instances, between PolarDB for MySQL clusters, between Tair (Enterprise Edition) instances, between ApsaraDB for MongoDB (replica set architecture) instances, and between ApsaraDB for MongoDB (sharded cluster architecture) instances.
NoteFor tasks that do not have the Replicate Data Across Alibaba Cloud Accounts configuration option, you can try to use Cloud Enterprise Network (CEN) to implement cross-account two-way synchronization tasks. For more information, see Access databases across accounts or regions.
Q: How do I configure parameters when ApsaraMQ for Kafka is the destination?
A: Configure the parameters based on your actual scenario. For the configuration methods of some special parameters, see Configure Kafka instance parameters.
Q: How many data synchronization instances can I try for free?
A: You can try one synchronization instance for free for a period of three months. For more information, see Alibaba Cloud Free Trial.
Q: How do I handle the "ERR invalid DB index" error when I use DTS to synchronize or migrate Redis data?
Cause: The
ERR invalid DB indexerror occurs when the destination database performs theSELECT DBoperation. This is typically caused by an insufficient number ofdatabasesin the destination database. If the destination database uses a proxy, check whether the proxy can remove the limit on the number ofdatabases.Solution: Modify the
databasesconfiguration of the destination database. We recommend that you increase the value to match that of the source database, and then restart the DTS task. The command to query thedatabasesconfiguration of the destination database is as follows:CONFIG GET databases;Q: How do I handle the "IDENTIFIER CLUSTERED" error when I synchronize or migrate data from SQL Server to AnalyticDB for PostgreSQL?
A:
Cause: When you synchronize or migrate SQL Server data to AnalyticDB for PostgreSQL, the task cannot proceed because the
CREATE CLUSTERED INDEXcommand is not supported.Solution: After you confirm that no other DDL commands exist in the latency period, Modify instance parameters set
sink.ignore.failed.ddlto true (to filter out the execution of all DDL statements). After the incremental synchronization or migration checkpoint advances, setsink.ignore.failed.ddlback to false.Q: In a Redis synchronization or migration task, does extending the expiration time of keys in the destination database have any actual effect?
A:
To prevent keys from expiring during full data synchronization, you can extend the expiration time of keys in the destination database in the task. For DTS tasks that include an incremental synchronization or migration part, if keys in the source database expire and are deleted, the corresponding keys in the destination database are also released.
Q: After the expiration time of keys in the destination database is extended, if a key in the source database expires, is the corresponding key in the destination database released immediately?
A:
Keys in the destination database are not necessarily released immediately upon expiration. A key in the destination database is released immediately only when the corresponding key in the source database expires and is cleaned up.
For example, if a key in the source database is set to expire in 5 seconds and the expiration time of the corresponding key in the destination database is extended to 30 seconds, when the source key expires and is automatically cleaned up after 5 seconds, the destination key is released immediately. When a key in the source database expires and is automatically cleaned up, the system appends a delete operation to the AOF file. The request is synchronized to the destination database and executed there.
Q: How do I handle the "column_name[xxx], the length of input string is too long than vec schema" error when I use DTS to synchronize data to SelectDB?
Cause: The incremental size of the xxx column at the source has exceeded the length limit of the xxx column type at the destination.
Solution: If the column is of the
STRINGtype, you can resolve the issue by increasing the string_type_length_soft_limit_bytes parameter in the SelectDB console. You must set it to a value greater than theactual length(7,241,898).
Q: How do I handle the "column(xxx) values is null while columns is not nullable." error when I use DTS to synchronize data to SelectDB?
Cause: The xxx column at the destination is
NOT NULL, butNULLvalues are written to it.Solution:
Change the column type at the destination to allow NULL values.
Modify the objects to synchronize: Remove the problematic table, correct the source data, and then drop the corresponding table at the destination. Modify the objects to synchronize again to add the table back, and then re-run full and incremental synchronization.
NoteCommon scenario: The source contains invalid date data such as
0000-0000-0000. DTS automatically converts such data toNULL, which triggers the error.Q: How do I correct an incorrectly configured destination table name mapping?
A: On the DTS task configuration page, in the Selected Objects section, expand the destination database, find the table with the incorrect mapping, and click the Edit button on the right. In the dialog box that appears, change the destination table name to the correct name, and then save the change.
Q: How do I backfill historical data after I add a new column?
DTS does not automatically backfill historical data for newly added columns. This rule applies to all destination database types, not just SelectDB. To backfill historical data, perform the following steps:
Manually add the new column to the destination table.
In the DTS console, modify the objects to synchronize and remove the table from the objects to synchronize.
Run
TRUNCATEto clear the data from the destination table (if you need to ensure data consistency).Add the table back to the objects to synchronize to trigger full data plus incremental data synchronization, which migrates the historical data, including the new column, to the destination database.
Q: When a DTS incremental synchronization task reports an error due to inconsistent table schemas, can I skip DML operations in bulk?
DTS does not support skipping DML operations in bulk directly. You can use one of the following two solutions:
Solution 1 (recommended): Add the missing DDL statements to make the table schemas of the source and destination consistent, and then restart the task. This solution does not cause data loss.
Solution 2: Modify the synchronization checkpoint (for example, skip a time range of one second or one minute) to skip the incremental records that caused the error.
WarningSolution 2 skips all DML operations (including INSERT, UPDATE, and DELETE) in that time range, which may cause data loss or inconsistency. Evaluate the impact carefully before you use this solution.
Q: What is the "Migrate Events" option in the DTS configuration? How do I choose a value for it?
The Migrate Events option controls whether to synchronize event objects in the source database to the destination database.
No (default): The event objects in the source database are not synchronized. If the destination database does not need to run the scheduled event logic, keep the default value.
Yes: The event definitions and trigger logic in the source database are synchronized to the destination database. If you need to fully reproduce the event scheduling of the source database in the destination database, select this option.
Q: How do I handle data inconsistency or schema desynchronization when I modify the objects to synchronize?
A: When any of the following issues occur, you can resolve them by removing and re-adding the objects to synchronize:
Data in the destination database is inconsistent with that in the source database.
Source-side DDL changes (such as adding a column or changing a column type) are not synchronized to the destination.
Tables with unique keys that contain NULL values are synchronized abnormally.
SQL syntax errors (such as the
near 'null'error).
Procedure:
Modify the objects to synchronize of the DTS task and remove the problematic table.
At the destination, drop (
DROP TABLE) or empty (TRUNCATE TABLE) the corresponding problematic table.Add the table back to the objects to synchronize of the DTS task, and start the precheck. DTS re-initializes the schema and performs full and incremental synchronization on the table.
Q: What is the impact of modifying data in the destination database while a data synchronization or migration task is running?
Modifying data in the destination database may cause the DTS task to fail. During data migration or synchronization, performing operations on the objects that are being migrated or synchronized in the destination database may cause issues such as primary key conflicts or missing update records, which ultimately cause the DTS task to fail. However, you can perform operations that do not interrupt the DTS task. For example, if you create a table in the destination instance and write data to it, the DTS task is not affected because the table is not included in the migration or synchronization objects.
Because DTS reads information from the source instance database and migrates or synchronizes its full data, schema data, and incremental data to the destination instance, data modifications made to the destination database during the task may be overwritten by data migrated or synchronized from the source database.
Q: When you use DTS for data synchronization or migration, is existing data at the destination overwritten?
A: Taking data synchronization as an example, the default behavior and related parameters of DTS are as follows:
Default behavior of DTS: If a primary key or unique key conflict occurs while the synchronization task is running:
If the table schemas are consistent and a record in the destination database has the same primary key or unique key value as a record in the source database:
During full data synchronization, DTS retains the destination record and skips the source record.
During incremental synchronization, DTS overwrites the destination record with the source record.
If the table schemas are inconsistent, data initialization may fail. This can result in only partial data synchronization or a complete synchronization failure. Use with caution.
Related parameters: When you configure a task, you can use the Processing Mode of Conflicting Tables parameter to manage how the related data is processed.
Precheck and Report Errors: Checks for tables with the same names in the destination database. If any tables with the same names are found, an error is reported during the precheck and the data synchronization task does not start. Otherwise, the precheck is successful.
NoteIf you cannot delete or rename the table with the same name in the destination database, you can map it to a different name in the destination. For more information, see Object name mapping.
Ignore Errors and Proceed: Skips the check for tables with the same name in the destination database.
WarningSelecting Ignore Errors and Proceed may cause data inconsistency and put your business at risk. For example:
If the table schemas are consistent and a record in the destination database has the same primary key or unique key value as a record in the source database:
During full data synchronization, DTS retains the destination record and skips the source record.
During incremental synchronization, DTS overwrites the destination record with the source record.
If the table schemas are inconsistent, data initialization may fail. This can result in only partial data synchronization or a complete synchronization failure. Use with caution.
Recommendation: To ensure data consistency, we recommend that you delete the destination table before configuring the DTS task, if your business allows.
Q: An incremental synchronization error indicates that a temporary table is missing in the destination database or that a SELECT INTO conflict occurs. How do I resolve this?
Cause:
This issue is not actually caused by a missing temporary table in the destination database. Instead, after Change Data Capture (CDC) is enabled in the source database, the triggers created by DTS intercept the implicit table creation performed by
SELECT INTO, which causes conflicts during data synchronization.Solution:
Verify that the transaction log file at the source exists and is intact.
Disable the CDC feature in the source database.
Log on to the DTS console, reconfigure the synchronization task, select Parse source database logs as the incremental synchronization method, and then restart the task.
Q: The synchronization task has been interrupted for several hours, and I suspect that an MDL lock exists at the destination. How do I resolve this?
Cause:
A metadata lock (MDL lock) at the destination, caused by a long-running DDL operation (such as adding a column to a large table or rebuilding an index), is blocking the incremental data writes from DTS, causing the synchronization link to be interrupted for a long time.
Solution:
Log on to the ApsaraDB RDS console, go to the destination database instance, and run
SHOW FULL PROCESSLISTto locate the abnormal session that holds the MDL lock. Then, kill the session.Log on to the DTS console and modify the synchronization objects to remove the tables that are affected by the MDL lock from the synchronization scope.
Drop the table in the destination database (before this operation, make sure that the data has been backed up or can be cleared).
After the DTS synchronization task returns to normal and the synchronization latency drops to zero, modify the synchronization objects again to add the table back to the synchronization scope and complete the re-migration.
Q: An incremental synchronization error indicates that the packet is too large or the max_allowed_packet value at the destination is insufficient. How do I resolve this?
Cause:
The source contains a row with a large amount of data that exceeds the limit of the
max_allowed_packetparameter at the destination during incremental synchronization, causing the synchronization error.Solution:
Log on to the source database and query the current value of the
max_allowed_packetparameter:SHOW VARIABLES LIKE 'max_allowed_packet';Record the value (for example,
1073741824, which is 1 GB).Log on to the ApsaraDB RDS console (or the PolarDB console if the destination is PolarDB for MySQL). On the Parameters page of the destination database instance, set the
max_allowed_packetparameter to a value that is no less than the value at the source.After the parameter modification is complete, restart the synchronization task in the DTS console and verify whether synchronization returns to normal.
Q: Can I adjust the rate after a synchronization or migration instance is running?
A: Yes. For more information, see Enable throttling for data migration.
Q: Is data in the source database retained after DTS synchronization or migration ends?
A: Yes. DTS does not delete data in the source database. If you do not need to retain the data in the source database, delete it manually.
Q: How do I use the API to configure the ETL feature for a synchronization or migration task?
A: You can configure it in the Reserve parameter of the API by using common parameters such as etlOperatorCtl and etlOperatorSetting. For more information, see ConfigureDtsJob and Reserve parameter reference.
Q: When synchronizing or migrating data, do the source database and the destination database need to be connected over the network?
A: No.
Q: When synchronizing or migrating data, do I need to manually create tables in the destination database?
A: For DTS instances that support schema tasks (schema synchronization or schema migration), if you have configured a schema task (Synchronization Types with Schema Synchronization selected, or Migration Types with Schema Migration selected), you do not need to manually create tables in the destination database.
Q: Does DTS support sampling data synchronization or migration by time range?
A: No.
Q: Can data be written to the source database and the destination database at the same time while a data synchronization or migration task is running?
A: Yes. However, while the DTS instance is running, if data sources other than DTS write data to the destination database, exceptions may occur in the destination database data or in the DTS instance.
Q: Does DTS support cross-account data migration or data synchronization?
A: Yes. For information about how to configure it, see Configure a cross-account DTS task.