Data Transmission Service:Precautions and limits for synchronizing data from a MySQL database

Category

Description

Limits on the source database

• The tables to synchronize must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

• If you select tables as the objects to be synchronized and you need to modify the tables, such as renaming tables or columns, you can synchronize up to 1,000 tables in a single data synchronization task. If you run a task to synchronize more than 1,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database.

• The following requirements for binary logs must be met:

  • By default, the binary logging feature is enabled. The binlog_row_image parameter must be set to full. Otherwise, error messages are returned during precheck and the data synchronization task cannot be started. For more information, see Modify instance parameters.

    Important

    • If the source database is a self-managed MySQL database, you must enable the binary logging feature and set the binlog_format parameter to row and the binlog_row_image parameter to full.

    • If the source database is a self-managed MySQL database deployed in a dual-primary cluster, you must set the log_slave_updates parameter to ON. This ensures that DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging.

  • The binary logs of the source database must be stored for more than seven days. Otherwise, DTS may fail to obtain the binary logs, which may cause the task to fail, or even data inconsistency and data loss. Make sure that you set the retention period of binary logs based on the preceding requirements. Otherwise, the service level agreement (SLA) of DTS does not guarantee service reliability or performance. For more information about how to manage the binary log files of an ApsaraDB RDS for MySQL instance, see Manage binary log files.

• During data synchronization, do not perform DDL operations to modify the primary key or add comments because the operations cannot take effect. For example, do not execute the ALTER TABLE table_name COMMENT='Table comments'; statement.

Other limits

• Prefix indexes cannot be synchronized. If the source database contains prefix indexes, data may fail to be synchronized.

• Due to the limits of AnalyticDB for MySQL clusters, if the disk space usage of the nodes in an AnalyticDB for MySQL cluster exceeds 80%, the data synchronization task is delayed and error messages are returned. We recommend that you estimate the required disk space based on the objects to be synchronized. Make sure that the destination cluster has sufficient storage space.

• Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads of the database servers.

• During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After initial full data synchronization is complete, the size of the used tablespace of the destination database is larger than that of the source database.

• If you select one or more tables instead of an entire database as the objects to be synchronized, do not use tools such as pt-online-schema-change to perform online DDL operations on the tables during data synchronization. Otherwise, data may fail to be synchronized.

  You can use Data Management (DMS) to perform online DDL operations. For more information, see Perform lock-free DDL operations.

• If you use only DTS to write data to the destination database in the AnalyticDB for MySQL cluster, you can use DMS to perform online DDL operations on source tables during data synchronization. For more information, see Perform lock-free DDL operations.

• During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. For example, if you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

• If DDL statements fail to be written to the destination database, the DTS task continues to run. You can view the DDL statements that fail to be executed in task logs. For more information about how to view the task logs, see View task logs.

Special cases

• If the source database is a self-managed MySQL database, take note of the following items:

  • If you perform a primary/secondary switchover on the source database when the data synchronization task is running, the task fails.

  • DTS calculates the synchronization latency based on the timestamp of the latest synchronized data in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for an extended period of time, the synchronization latency may be inaccurate. If the synchronization latency is too high, you can perform a DML operation on the source database to update the latency.

    Note

    If you select an entire database as the object to be synchronized, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

  • DTS executes the CREATE DATABASE IF NOT EXISTS 'test' statement in the source database as scheduled to move forward the binary log file position.

• If the source database is an Apsara RDS for MySQL instance, take note of the following item:

  The source Apsara RDS for MySQL instance cannot be a read-only ApsaraDB RDS for MySQL instance that does not record transaction logs, such as an ApsaraDB RDS for MySQL V5.6 instance.

Category

Description

Limits on the source database

• The tables to synchronize must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

• If you select tables as the objects to be synchronized and you need to modify the tables, such as renaming tables or columns, you can synchronize up to 1,000 tables in a single data synchronization task. If you run a task to synchronize more than 1,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database.

• The following requirements for binary logs must be met:

  • By default, the binary logging feature is enabled. The binlog_row_image parameter must be set to full. Otherwise, error messages are returned during precheck and the data synchronization task cannot be started. For more information, see Modify instance parameters.

    Important

    • If the source database is a self-managed MySQL database, you must enable the binary logging feature and set the binlog_format parameter to row and the binlog_row_image parameter to full.

    • If the source database is a self-managed MySQL database deployed in a dual-primary cluster, you must set the log_slave_updates parameter to ON. This ensures that DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging.

  • The binary logs of the source database must be stored for more than seven days. Otherwise, DTS may fail to obtain the binary logs, which may cause the task to fail, or even data inconsistency and data loss. Make sure that you set the retention period of binary logs based on the preceding requirements. Otherwise, the SLA of DTS does not guarantee service reliability or performance. For more information about how to manage the binary log files of an ApsaraDB RDS for MySQL instance, see Manage binary log files.

• During data synchronization, do not perform DDL operations to modify the primary key or add comments because the operations cannot take effect. For example, do not execute the ALTER TABLE table_name COMMENT='Table comments'; statement.

Other limits

• Requirements for the objects to be synchronized:

  • Only tables can be selected as the objects to be synchronized.

  • DTS does not synchronize data of the following data types: BIT, VARBIT, GEOMETRY, ARRAY, UUID, TSQUERY, TSVECTOR, TXID_SNAPSHOT, and POINT.

  • Prefix indexes cannot be synchronized. If the source database contains prefix indexes, data may fail to be synchronized.

• Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads of the database servers.

• During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After initial full data synchronization is complete, the size of the used tablespace of the destination database is larger than that of the source database.

• If you select one or more tables instead of an entire database as the objects to be synchronized, do not use tools such as pt-online-schema-change to perform online DDL operations on the tables during data synchronization. Otherwise, data may fail to be synchronized.

  You can use Data Management (DMS) to perform online DDL operations. For more information, see Perform lock-free DDL operations.

• During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. For example, if you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

• Append-optimized (AO) tables cannot be used as destination tables.

• If column mapping is used for non-full table synchronization or if the source and destination table schemas are inconsistent, the data is lost for the columns that are not contained in the tables of the destination database.

Special cases

• If the source database is a self-managed MySQL database, take note of the following items:

  • If you perform a primary/secondary switchover on the source database when the data synchronization task is running, the task fails.

  • DTS calculates the synchronization latency based on the timestamp of the latest synchronized data in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for an extended period of time, the synchronization latency may be inaccurate. If the synchronization latency is too high, you can perform a DML operation on the source database to update the latency.

    Note

    If you select an entire database as the object to be synchronized, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

  • DTS executes the CREATE DATABASE IF NOT EXISTS 'test' statement in the source database as scheduled to move forward the binary log file position.

• If the source database is an Apsara RDS for MySQL instance, take note of the following item:

  The source Apsara RDS for MySQL instance cannot be a read-only ApsaraDB RDS for MySQL instance that does not record transaction logs, such as an ApsaraDB RDS for MySQL V5.6 instance.

Category

Description

Limits on the source database

• The tables to synchronize must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

• If you select tables as the objects to be synchronized and you need to modify the tables, such as renaming tables or columns, you can synchronize up to 1,000 tables in a single data synchronization task. If you run a task to synchronize more than 1,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database.

• The following requirements for binary logs must be met:

  • By default, the binary logging feature is enabled. The binlog_row_image parameter must be set to full. Otherwise, error messages are returned during precheck and the data synchronization task cannot be started. For more information, see Modify instance parameters.

    Important

    • If the source database is a self-managed MySQL database, you must enable the binary logging feature and set the binlog_format parameter to row and the binlog_row_image parameter to full.

    • If the source database is a self-managed MySQL database deployed in a dual-primary cluster, you must set the log_slave_updates parameter to ON. This ensures that DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging.

  • The binary logs of the source database must be stored for more than seven days. Otherwise, DTS may fail to obtain the binary logs, which may cause the task to fail, or even data inconsistency and data loss. Make sure that you set the retention period of binary logs based on the preceding requirements. Otherwise, the SLA of DTS does not guarantee service reliability or performance. For more information about how to manage the binary log files of an ApsaraDB RDS for MySQL instance, see Manage binary log files.

Other limits

• Initial full data synchronization is not supported. DTS does not synchronize the historical data of the required objects from the source ApsaraDB RDS for MySQL instance to the destination DataHub project.

• A single string in the destination DataHub project cannot exceed 2 MB in length.

• If a table to be synchronized in the source database has a column named record_id, we recommend that you use the object name mapping feature to rename the column in the destination database. Otherwise, an error message is returned. For more information, see Map object names.

• During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. For example, if you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

Special cases

• If the source database is a self-managed MySQL database, take note of the following items:

  • If you perform a primary/secondary switchover on the source database when the data synchronization task is running, the task fails.

  • DTS calculates the synchronization latency based on the timestamp of the latest synchronized data in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for an extended period of time, the synchronization latency may be inaccurate. If the synchronization latency is too high, you can perform a DML operation on the source database to update the latency.

    Note

    If you select an entire database as the object to be synchronized, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

  • DTS executes the CREATE DATABASE IF NOT EXISTS 'test' statement in the source database as scheduled to move forward the binary log file position.

• If the source database is an Apsara RDS for MySQL instance, take note of the following item:

  The source Apsara RDS for MySQL instance cannot be a read-only ApsaraDB RDS for MySQL instance that does not record transaction logs, such as an ApsaraDB RDS for MySQL V5.6 instance.

Category

Description

Limits on the source database

• The tables to synchronize must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

• If you select tables as the objects to be synchronized and you need to modify the tables, such as renaming tables or columns, you can synchronize up to 1,000 tables in a single data synchronization task. If you run a task to synchronize more than 1,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database.

• The following requirements for binary logs must be met:

  • By default, the binary logging feature is enabled. The binlog_row_image parameter must be set to full. Otherwise, error messages are returned during precheck and the data synchronization task cannot be started. For more information, see Modify instance parameters.

    Important

    • If the source database is a self-managed MySQL database, you must enable the binary logging feature and set the binlog_format parameter to row and the binlog_row_image parameter to full.

    • If the source database is a self-managed MySQL database deployed in a dual-primary cluster, you must set the log_slave_updates parameter to ON. This ensures that DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging.

  • The binary logs of the source database must be stored for more than seven days. Otherwise, DTS may fail to obtain the binary logs, which may cause the task to fail, or even data inconsistency and data loss. Make sure that you set the retention period of binary logs based on the preceding requirements. Otherwise, the service level agreement (SLA) of DTS does not guarantee service reliability or performance. For more information about how to manage the binary log files of an ApsaraDB RDS for MySQL instance, see Manage binary log files.

Other limits

• Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads of the database servers.

• During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After initial full data synchronization is complete, the size of the used tablespace of the destination database is larger than that of the source database.

• If you select one or more tables instead of an entire database as the objects to be synchronized, do not use tools such as pt-online-schema-change to perform online DDL operations on the tables during data synchronization. Otherwise, data may fail to be synchronized.

  You can use Data Management (DMS) to perform online DDL operations. For more information, see Perform lock-free DDL operations.

• To add columns to a table that you want to synchronize, perform the following operations: Modify the mappings of the table in the Elasticsearch cluster, perform DDL operations in the source MySQL database, and then pause and start the data synchronization task.

• During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. For example, if you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

• If a table that you want to synchronize contains empty strings, the data that contains empty strings in the table is written to the Elasticsearch cluster as data of the LONG type. In this case, the data synchronization task fails.

• If a table that you want to synchronize contains latitude and longitude values that are stored in reverse, an error occurs when the data in the table is written to the Elasticsearch cluster.

Special cases

• If the source database is a self-managed MySQL database, take note of the following items:

  • If you perform a primary/secondary switchover on the source database when the data synchronization task is running, the task fails.

  • DTS calculates the synchronization latency based on the timestamp of the latest synchronized data in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for an extended period of time, the synchronization latency may be inaccurate. If the synchronization latency is too high, you can perform a DML operation on the source database to update the latency.

    Note

    If you select an entire database as the object to be synchronized, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

  • DTS executes the CREATE DATABASE IF NOT EXISTS 'test' statement in the source database as scheduled to move forward the binary log file position.

• If the source database is an Apsara RDS for MySQL instance, take note of the following item:

  The source Apsara RDS for MySQL instance cannot be a read-only ApsaraDB RDS for MySQL instance that does not record transaction logs, such as an ApsaraDB RDS for MySQL V5.6 instance.

Category

Description

Limits on the source database

• The tables to synchronize must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

• If you select tables as the objects to be synchronized and you need to modify the tables, such as renaming tables or columns, you can synchronize up to 1,000 tables in a single data synchronization task. If you run a task to synchronize more than 1,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database.

• The following requirements for binary logs must be met:

  • By default, the binary logging feature is enabled. The binlog_row_image parameter must be set to full. Otherwise, error messages are returned during precheck and the data synchronization task cannot be started. For more information, see Modify instance parameters.

    Important

    • If the source database is a self-managed MySQL database, you must enable the binary logging feature and set the binlog_format parameter to row and the binlog_row_image parameter to full.

    • If the source database is a self-managed MySQL database deployed in a dual-primary cluster, you must set the log_slave_updates parameter to ON. This ensures that DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging.

  • The binary logs of the source database must be stored for more than seven days. Otherwise, DTS may fail to obtain the binary logs, which may cause the task to fail, or even data inconsistency and data loss. Make sure that you set the retention period of binary logs based on the preceding requirements. Otherwise, the service level agreement (SLA) of DTS does not guarantee service reliability or performance. For more information about how to manage the binary log files of an ApsaraDB RDS for MySQL instance, see Manage binary log files.

Other limits

• Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads of the database servers.

• During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After initial full data synchronization is complete, the size of the used tablespace of the destination database is larger than that of the source database.

• If you select one or more tables instead of an entire database as the objects to be synchronized, do not use tools such as pt-online-schema-change to perform online DDL operations on the tables during data synchronization. Otherwise, data may fail to be synchronized.

  You can use Data Management (DMS) to perform online DDL operations. For more information, see Perform lock-free DDL operations.

• MaxCompute does not support PRIMARY KEY constraints. If network errors occur, DTS may synchronize duplicate data records to the MaxCompute project.

• During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. For example, if you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

Special cases

• If the source database is a self-managed MySQL database, take note of the following items:

  • If you perform a primary/secondary switchover on the source database when the data synchronization task is running, the task fails.

  • DTS calculates the synchronization latency based on the timestamp of the latest synchronized data in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for an extended period of time, the synchronization latency may be inaccurate. If the synchronization latency is too high, you can perform a DML operation on the source database to update the latency.

    Note

    If you select an entire database as the object to be synchronized, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

  • DTS executes the CREATE DATABASE IF NOT EXISTS 'test' statement in the source database as scheduled to move forward the binary log file position.

• If the source database is an Apsara RDS for MySQL instance, take note of the following item:

  The source Apsara RDS for MySQL instance cannot be a read-only ApsaraDB RDS for MySQL instance that does not record transaction logs, such as an ApsaraDB RDS for MySQL V5.6 instance.

Category

Description

Limits on the source database

• The tables to synchronize must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

• If you select tables as the objects to be synchronized and you need to modify the tables, such as renaming tables or columns, you can synchronize up to 1,000 tables in a single data synchronization task. If you run a task to synchronize more than 1,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database.

• The following requirements for binary logs must be met:

  • By default, the binary logging feature is enabled. The binlog_row_image parameter must be set to full. Otherwise, error messages are returned during precheck and the data synchronization task cannot be started. For more information, see Modify instance parameters.

    Important

    • If the source database is a self-managed MySQL database, you must enable the binary logging feature and set the binlog_format parameter to row and the binlog_row_image parameter to full.

    • If the source database is a self-managed MySQL database deployed in a dual-primary cluster, you must set the log_slave_updates parameter to ON. This ensures that DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging.

  • The binary logs of the source database must be stored for more than seven days. Otherwise, DTS may fail to obtain the binary logs, which may cause the task to fail, or even data inconsistency and data loss. Make sure that you set the retention period of binary logs based on the preceding requirements. Otherwise, the service level agreement (SLA) of DTS does not guarantee service reliability or performance. For more information about how to manage the binary log files of an ApsaraDB RDS for MySQL instance, see Manage binary log files.

Other limits

• Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads of the database servers.

• During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After initial full data synchronization is complete, the size of the used tablespace of the destination database is larger than that of the source database.

• If you select one or more tables instead of an entire database as the objects to be synchronized, do not use tools such as pt-online-schema-change to perform online DDL operations on the tables during data synchronization. Otherwise, data may fail to be synchronized.

  You can use Data Management (DMS) to perform online DDL operations. For more information, see Perform lock-free DDL operations.

• During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. For example, if you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

• The names of the databases, tables, and columns to be synchronized must comply with the naming conventions of ApsaraDB for ClickHouse. For more information, see the "Object naming conventions" section of the Limits topic.

Special cases

• If the source database is a self-managed MySQL database, take note of the following items:

  • If you perform a primary/secondary switchover on the source database when the data synchronization task is running, the task fails.

  • DTS calculates the synchronization latency based on the timestamp of the latest synchronized data in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for an extended period of time, the synchronization latency may be inaccurate. If the synchronization latency is too high, you can perform a DML operation on the source database to update the latency.

    Note

    If you select an entire database as the object to be synchronized, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

  • DTS executes the CREATE DATABASE IF NOT EXISTS 'test' statement in the source database as scheduled to move forward the binary log file position.

• If the source database is an Apsara RDS for MySQL instance, take note of the following item:

  The source Apsara RDS for MySQL instance cannot be a read-only ApsaraDB RDS for MySQL instance that does not record transaction logs, such as an ApsaraDB RDS for MySQL V5.6 instance.

Category

Description

Limits on the source database

• The tables to synchronize must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

• If you select tables as the objects to be synchronized and you need to modify the tables, such as renaming tables or columns, you can synchronize up to 1,000 tables in a single data synchronization task. If you run a task to synchronize more than 1,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database.

• The following requirements for binary logs must be met:

  • By default, the binary logging feature is enabled. The binlog_row_image parameter must be set to full. Otherwise, error messages are returned during precheck and the data synchronization task cannot be started. For more information, see Modify instance parameters.

    Important

    • If the source database is a self-managed MySQL database, you must enable the binary logging feature and set the binlog_format parameter to row and the binlog_row_image parameter to full.

    • If the source database is a self-managed MySQL database deployed in a dual-primary cluster, you must set the log_slave_updates parameter to ON. This ensures that DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging.

  • The binary logs of the source database must be stored for more than seven days. Otherwise, DTS may fail to obtain the binary logs, which may cause the task to fail, or even data inconsistency and data loss. Make sure that you set the retention period of binary logs based on the preceding requirements. Otherwise, the service level agreement (SLA) of DTS does not guarantee service reliability or performance. For more information about how to manage the binary log files of an ApsaraDB RDS for MySQL instance, see Manage binary log files.

Other limits

• Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads of the database servers.

• During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After initial full data synchronization is complete, the size of the used tablespace of the destination database is larger than that of the source database.

• If you select one or more tables instead of an entire database as the objects to be synchronized, do not use tools such as pt-online-schema-change to perform online DDL operations on the tables during data synchronization. Otherwise, data may fail to be synchronized.

  You can use Data Management (DMS) to perform online DDL operations. For more information, see Perform lock-free DDL operations.

• During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. For example, if you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

• The names of the tables and columns to be synchronized must comply with the naming conventions of the Tablestore instance.

  • The name of a table or column can contain letters, digits, and underscores (_). The name must start with a letter or an underscore (_).

  • The name of a table or column must be 1 to 255 characters in length.

Special cases

• If the source database is a self-managed MySQL database, take note of the following items:

  • If you perform a primary/secondary switchover on the source database when the data synchronization task is running, the task fails.

  • DTS calculates the synchronization latency based on the timestamp of the latest synchronized data in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for an extended period of time, the synchronization latency may be inaccurate. If the synchronization latency is too high, you can perform a DML operation on the source database to update the latency.

    Note

    If you select an entire database as the object to be synchronized, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

  • DTS executes the CREATE DATABASE IF NOT EXISTS 'test' statement in the source database as scheduled to move forward the binary log file position.

• If the source database is an Apsara RDS for MySQL instance, take note of the following item:

  The source Apsara RDS for MySQL instance cannot be a read-only ApsaraDB RDS for MySQL instance that does not record transaction logs, such as an ApsaraDB RDS for MySQL V5.6 instance.

Category

Description

Limits on the source database

• The tables to synchronize must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

• If you select tables as the objects to be synchronized and you need to modify the tables, such as renaming tables or columns, you can synchronize up to 1,000 tables in a single data synchronization task. If you run a task to synchronize more than 1,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database.

• The following requirements for binary logs must be met:

  • By default, the binary logging feature is enabled. The binlog_row_image parameter must be set to full. Otherwise, error messages are returned during precheck and the data synchronization task cannot be started. For more information, see Modify instance parameters.

    Important

    • If the source database is a self-managed MySQL database, you must enable the binary logging feature and set the binlog_format parameter to row and the binlog_row_image parameter to full.

    • If the source database is a self-managed MySQL database deployed in a dual-primary cluster, you must set the log_slave_updates parameter to ON. This ensures that DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging.

  • The binary logs of the source database must be stored for more than seven days. Otherwise, DTS may fail to obtain the binary logs, which may cause the task to fail, or even data inconsistency and data loss. Make sure that you set the retention period of binary logs based on the preceding requirements. Otherwise, the service level agreement (SLA) of DTS does not guarantee service reliability or performance. For more information about how to manage the binary log files of an ApsaraDB RDS for MySQL instance, see Manage binary log files.

Other limits

• Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads of the database servers.

• During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After initial full data synchronization is complete, the size of the used tablespace of the destination database is larger than that of the source database.

• If you select one or more tables instead of an entire database as the objects to be synchronized, do not use tools such as pt-online-schema-change to perform online DDL operations on the tables during data synchronization. Otherwise, data may fail to be synchronized.

  You can use Data Management (DMS) to perform online DDL operations. For more information, see Perform lock-free DDL operations.

• During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. For example, if you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

• During data synchronization, if the destination Kafka instance or cluster is scaled, you must restart the instance or cluster.

Special cases

• If the source database is a self-managed MySQL database, take note of the following items:

  • If you perform a primary/secondary switchover on the source database when the data synchronization task is running, the task fails.

  • DTS calculates the synchronization latency based on the timestamp of the latest synchronized data in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for an extended period of time, the synchronization latency may be inaccurate. If the synchronization latency is too high, you can perform a DML operation on the source database to update the latency.

    Note

    If you select an entire database as the object to be synchronized, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

  • DTS executes the CREATE DATABASE IF NOT EXISTS 'test' statement in the source database as scheduled to move forward the binary log file position.

• If the source database is an Apsara RDS for MySQL instance, take note of the following item:

  The source Apsara RDS for MySQL instance cannot be a read-only ApsaraDB RDS for MySQL instance that does not record transaction logs, such as an ApsaraDB RDS for MySQL V5.6 instance.

Category

Description

Limits on the source database

• The tables to synchronize must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

• If you select tables as the objects to be synchronized and you need to modify the tables, such as renaming tables or columns, you can synchronize up to 1,000 tables in a single data synchronization task. If you run a task to synchronize more than 1,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database.

• The following requirements for binary logs must be met:

  • By default, the binary logging feature is enabled. The binlog_row_image parameter must be set to full. Otherwise, error messages are returned during precheck and the data synchronization task cannot be started. For more information, see Modify instance parameters.

    Important

    • If the source database is a self-managed MySQL database, you must enable the binary logging feature and set the binlog_format parameter to row and the binlog_row_image parameter to full.

    • If the source database is a self-managed MySQL database deployed in a dual-primary cluster, you must set the log_slave_updates parameter to ON. This ensures that DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging.

  • The binary logs of the source database must be stored for more than seven days. Otherwise, DTS may fail to obtain the binary logs, which may cause the task to fail, or even data inconsistency and data loss. Make sure that you set the retention period of binary logs based on the preceding requirements. Otherwise, the service level agreement (SLA) of DTS does not guarantee service reliability or performance. For more information about how to manage the binary log files of an ApsaraDB RDS for MySQL instance, see Manage binary log files.

Other limits

• Requirements for the objects to be synchronized:

  • DTS does not synchronize data of the following data types: BIT, VARBIT, GEOMETRY, ARRAY, UUID, TSQUERY, TSVECTOR, and TXID_SNAPSHOT.

  • Prefix indexes cannot be synchronized. If the source database contains prefix indexes, data may fail to be synchronized.

• Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads of the database servers.

• During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After initial full data synchronization is complete, the size of the used tablespace of the destination database is larger than that of the source database.

• If you select one or more tables instead of an entire database as the objects to be synchronized, do not use tools such as pt-online-schema-change to perform online DDL operations on the tables during data synchronization. Otherwise, data may fail to be synchronized.

  You can use Data Management (DMS) to perform online DDL operations. For more information, see Perform lock-free DDL operations.

• During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. For example, if you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

Special cases

• If the source database is a self-managed MySQL database, take note of the following items:

  • If you perform a primary/secondary switchover on the source database when the data synchronization task is running, the task fails.

  • DTS calculates the synchronization latency based on the timestamp of the latest synchronized data in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for an extended period of time, the synchronization latency may be inaccurate. If the synchronization latency is too high, you can perform a DML operation on the source database to update the latency.

    Note

    If you select an entire database as the object to be synchronized, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

  • DTS executes the CREATE DATABASE IF NOT EXISTS 'test' statement in the source database as scheduled to move forward the binary log file position.

• If the source database is an Apsara RDS for MySQL instance, take note of the following item:

  The source Apsara RDS for MySQL instance cannot be a read-only ApsaraDB RDS for MySQL instance that does not record transaction logs, such as an ApsaraDB RDS for MySQL V5.6 instance.

Category

Description

Limits on the source database

• The tables to synchronize must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

• If you select tables as the objects to be synchronized and you need to modify the tables, such as renaming tables or columns, you can synchronize up to 1,000 tables in a single data synchronization task. If you run a task to synchronize more than 1,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database.

• The following requirements for binary logs must be met:

  • By default, the binary logging feature is enabled. The binlog_row_image parameter must be set to full. Otherwise, error messages are returned during precheck and the data synchronization task cannot be started. For more information, see Modify instance parameters.

    Important

    • If the source database is a self-managed MySQL database, you must enable the binary logging feature and set the binlog_format parameter to row and the binlog_row_image parameter to full.

    • If the source database is a self-managed MySQL database deployed in a dual-primary cluster, you must set the log_slave_updates parameter to ON. This ensures that DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging.

  • The binary logs of the source database must be stored for more than seven days. Otherwise, DTS may fail to obtain the binary logs, which may cause the task to fail, or even data inconsistency and data loss. Make sure that you set the retention period of binary logs based on the preceding requirements. Otherwise, the SLA of DTS does not guarantee service reliability or performance. For more information about how to manage the binary log files of an ApsaraDB RDS for MySQL instance, see Manage binary log files.

Other limits

• The attributes of at most one column can be changed.

• The basic data types cannot be changed.

• If the self-managed Oracle database is an Oracle Real Application Cluster (RAC) database, you can only use a single virtual IP address (VIP) rather than a Single Client Access Name (SCAN) IP address when you configure the data synchronization task. After you specify the VIP, node failover is not supported for the Oracle RAC database.

Special cases

• If the source database is a self-managed MySQL database, take note of the following items:

  • If you perform a primary/secondary switchover on the source database when the data synchronization task is running, the task fails.

  • DTS calculates the synchronization latency based on the timestamp of the latest synchronized data in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for an extended period of time, the synchronization latency may be inaccurate. If the synchronization latency is too high, you can perform a DML operation on the source database to update the latency.

    Note

    If you select an entire database as the object to be synchronized, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

  • DTS executes the CREATE DATABASE IF NOT EXISTS 'test' statement in the source database as scheduled to move forward the binary log file position.

• If the source database is an Apsara RDS for MySQL instance, take note of the following item:

  The source Apsara RDS for MySQL instance cannot be a read-only ApsaraDB RDS for MySQL instance that does not record transaction logs, such as an ApsaraDB RDS for MySQL V5.6 instance.

Category

Description

Limits on the source database

• The tables to synchronize must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

• If you select tables as the objects to be synchronized and you need to modify the tables, such as renaming tables or columns, you can synchronize up to 1,000 tables in a single data synchronization task. If you run a task to synchronize more than 1,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database.

• The following requirements for binary logs must be met:

  • By default, the binary logging feature is enabled. The binlog_row_image parameter must be set to full. Otherwise, error messages are returned during precheck and the data synchronization task cannot be started. For more information, see Modify instance parameters.

    Important

    • If the source database is a self-managed MySQL database, you must enable the binary logging feature and set the binlog_format parameter to row and the binlog_row_image parameter to full.

    • If the source database is a self-managed MySQL database deployed in a dual-primary cluster, you must set the log_slave_updates parameter to ON. This ensures that DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging.

  • The binary logs of the source database must be stored for more than seven days. Otherwise, DTS may fail to obtain the binary logs, which may cause the task to fail, or even data inconsistency and data loss. Make sure that you set the retention period of binary logs based on the preceding requirements. Otherwise, the SLA of DTS does not guarantee service reliability or performance. For more information about how to manage the binary log files of an ApsaraDB RDS for MySQL instance, see Manage binary log files.

Other limits

• Requirements for the objects to be synchronized:

  • Only tables can be selected as the objects to be synchronized.

  • DTS does not synchronize data of the following data types: BIT, VARBIT, GEOMETRY, ARRAY, UUID, TSQUERY, TSVECTOR, TXID_SNAPSHOT, and POINT.

  • Prefix indexes cannot be synchronized. If the source database contains prefix indexes, data may fail to be synchronized.

• Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads of the database servers.

• During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After initial full data synchronization is complete, the size of the used tablespace of the destination database is larger than that of the source database.

• If you select one or more tables instead of an entire database as the objects to be synchronized, do not use tools such as pt-online-schema-change to perform online DDL operations on the tables during data synchronization. Otherwise, data may fail to be synchronized.

  You can use Data Management (DMS) to perform online DDL operations. For more information, see Perform lock-free DDL operations.

• During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. For example, if you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

Special cases

• If the source database is a self-managed MySQL database, take note of the following items:

  • If you perform a primary/secondary switchover on the source database when the data synchronization task is running, the task fails.

  • DTS calculates the synchronization latency based on the timestamp of the latest synchronized data in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for an extended period of time, the synchronization latency may be inaccurate. If the synchronization latency is too high, you can perform a DML operation on the source database to update the latency.

    Note

    If you select an entire database as the object to be synchronized, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

  • DTS executes the CREATE DATABASE IF NOT EXISTS 'test' statement in the source database as scheduled to move forward the binary log file position.

• If the source database is an Apsara RDS for MySQL instance, take note of the following item:

  The source Apsara RDS for MySQL instance cannot be a read-only ApsaraDB RDS for MySQL instance that does not record transaction logs, such as an ApsaraDB RDS for MySQL V5.6 instance.