Overview of synchronization scenarios for a PolarDB for MySQL source instance
Review the notes and limits for your synchronization scenario:
Note By default, DTS disables foreign key constraints when it synchronizes data to a destination database. Therefore, operations such as cascade and delete in the source database are not synchronized to the following types of destination databases:
Synchronization between PolarDB for MySQL clusters
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not execute DDL statements to change the schemas of databases or tables. Otherwise, the data synchronization task fails.
|
Other limits | You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Synchronization of INDEX and PARTITION is not supported. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If online DDL operations that use temporary tables, such as merging multiple tables, are performed on the source database, data may be lost in the destination database or the synchronization instance may fail. If a primary key or unique key conflict occurs while the synchronization instance is running: If the source and destination databases have the same schema and a data record in the destination database has the same primary key value or unique key value as a data record in the source database: During full data synchronization, DTS does not synchronize the data record to the destination database. The existing data record in the destination database is retained. During incremental data synchronization, DTS synchronizes the data record to the destination database. The existing data record in the destination database is overwritten.
If the source and destination databases have different schemas, data may fail to be initialized. In this case, only some columns are synchronized, or the data synchronization instance fails. Proceed with caution.
Synchronization of parsers defined using comment syntax is not supported. If the data to be synchronized contains information such as rare characters or emojis that takes up four bytes, the destination databases and tables to receive the data must use UTF8mb4 character set.
Note If you use the schema synchronization feature of DTS, set the instance parameter character_set_server in the destination database to UTF8mb4 character set. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If a DDL operation fails to be written to the destination database, the DTS task continues to run. You need to check the task logs for the failed DDL operation. For more information about how to view task logs, see View task logs. To synchronize accounts from the source database, you must also meet the prerequisites and understand the related notes. For more information, see Migrate database accounts. If a DTS task fails to run, DTS technical support will try to restore the task within 8 hours. During the restoration, the task may be restarted, and the parameters of the task may be modified.
Note Only the parameters of the DTS task may be modified. The parameters of databases are not modified. The parameters that may be modified include but are not limited to the parameters in the "Modify instance parameters" section of the Modify the parameters of a DTS instance topic.
|
Other notes | Two-way synchronization between PolarDB for MySQL clusters: DTS currently supports two-way synchronization between only two PolarDB for MySQL clusters. Two-way synchronization among multiple PolarDB for MySQL clusters is not supported. DDL syntax synchronization direction limit. To ensure the stability and data consistency of the two-way synchronization link, only forward DDL synchronization is supported. Reverse DDL synchronization is not supported. When a two-way synchronization task is running, DTS creates a database named DTS in the destination databases of the forward and reverse tasks to prevent data loop synchronization. Do not modify this database while the task is running, and make sure that the database account used for the task has read and write permissions on this database. A two-way data synchronization instance contains a forward synchronization task and a reverse synchronization task. If an object is to be synchronized in both the forward and reverse synchronization tasks when you configure or reset the instance, the following rules apply: Only one of the tasks can synchronize both the full data and incremental data of objects. The other task synchronizes only the incremental data of the objects. The source data of the current task can be synchronized only to the destination database in the task. The synchronized data is not used as the source data of the other task.
DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset.
|
Synchronization from PolarDB for MySQL to ApsaraDB RDS for MySQL or self-managed MySQL
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not execute DDL statements to change the schemas of databases or tables. Otherwise, the data synchronization task fails.
|
Other limits | You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Synchronization of INDEX and PARTITION is not supported. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If online DDL operations that use temporary tables, such as merging multiple tables, are performed on the source database, data may be lost in the destination database or the synchronization instance may fail. Synchronization of parsers defined using comment syntax is not supported. If a primary key or unique key conflict occurs while the synchronization instance is running: If the source and destination databases have the same schema and a data record in the destination database has the same primary key value or unique key value as a data record in the source database: During full data synchronization, DTS does not synchronize the data record to the destination database. The existing data record in the destination database is retained. During incremental data synchronization, DTS synchronizes the data record to the destination database. The existing data record in the destination database is overwritten.
If the source and destination databases have different schemas, data may fail to be initialized. In this case, only some columns are synchronized, or the data synchronization instance fails. Proceed with caution.
If the data to be synchronized contains information such as rare characters or emojis that takes up four bytes, the destination databases and tables to receive the data must use UTF8mb4 character set.
Note If you use the schema synchronization feature of DTS, set the instance parameter character_set_server in the destination database to UTF8mb4 character set. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If a DDL operation fails to be written to the destination database, the DTS task continues to run. You need to check the task logs for the failed DDL operation. For more information about how to view task logs, see View task logs. Column names in MySQL databases are not case-sensitive. Therefore, if multiple columns in the source database have names that differ only in capitalization, data in the columns are written to the same column in the destination MySQL database during the synchronization. This can cause unexpected synchronization results. After data synchronization is complete, that is, the Status of the instance changes to Completed, we recommend that you run the analyze table <Table name> command to check whether data is written to the destination table. For example, if a high-availability (HA) switchover is triggered in the source MySQL database, data may be written only to the memory. As a result, data loss occurs. A two-way data synchronization instance contains a forward synchronization task and a reverse synchronization task. If an object is to be synchronized in both the forward and reverse synchronization tasks when you configure or reset the instance, the following rules apply: Only one of the tasks can synchronize both the full data and incremental data of objects. The other task synchronizes only the incremental data of the objects. The source data of the current task can be synchronized only to the destination database in the task. The synchronized data is not used as the source data of the other task.
If a DTS task fails to run, DTS technical support will try to restore the task within 8 hours. During the restoration, the task may be restarted, and the parameters of the task may be modified.
Note Only the parameters of the DTS task may be modified. The parameters of databases are not modified. The parameters that may be modified include but are not limited to the parameters in the "Modify instance parameters" section of the Modify the parameters of a DTS instance topic.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to PolarDB-X 1.0
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During full data synchronization, do not perform DDL operations to change the schemas of databases or tables. Otherwise, the data synchronization task fails.
|
Other limits | Synchronization object requirements: Synchronization of data of the BIT, VARBIT, GEOMETRY, ARRAY, UUID, TSQUERY, TSVECTOR, and TXID_SNAPSHOT types is not supported. Synchronization of prefix indexes is not supported. If the source database has prefix indexes, the data synchronization may fail. You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster.
Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If online DDL operations that use temporary tables, such as merging multiple tables, are performed on the source database, data may be lost in the destination database or the synchronization instance may fail. Initial schema synchronization is not supported. Before you configure the synchronization task, you must create the corresponding databases and tables in the destination instance. If a primary key or unique key conflict occurs while the synchronization instance is running: If the source and destination databases have the same schema and a data record in the destination database has the same primary key value or unique key value as a data record in the source database: During full data synchronization, DTS does not synchronize the data record to the destination database. The existing data record in the destination database is retained. During incremental data synchronization, DTS synchronizes the data record to the destination database. The existing data record in the destination database is overwritten.
If the source and destination databases have different schemas, data may fail to be initialized. In this case, only some columns are synchronized, or the data synchronization instance fails. Proceed with caution.
Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If a DTS task fails to run, DTS technical support will try to restore the task within 8 hours. During the restoration, the task may be restarted, and the parameters of the task may be modified.
Note Only the parameters of the DTS task may be modified. The parameters of databases are not modified. The parameters that may be modified include but are not limited to the parameters in the "Modify instance parameters" section of the Modify the parameters of a DTS instance topic.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to AnalyticDB for MySQL
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During synchronization, do not perform DDL operations to modify primary keys or add comments to tables, such as ALTER TABLE table_name COMMENT='Table comment';. Otherwise, the DDL operations fail to be executed during data synchronization. During schema synchronization and full data synchronization, do not execute DDL statements to change the schemas of databases or tables. Otherwise, the data synchronization task fails.
|
Other limits | Synchronization of prefix indexes is not supported. If the source database has prefix indexes, the data synchronization may fail. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If online DDL operations that use temporary tables, such as merging multiple tables, are performed on the source database, data may be lost in the destination database or the synchronization instance may fail. If a primary key or unique key conflict occurs while the synchronization instance is running: If the source and destination databases have the same schema and a data record in the destination database has the same primary key value or unique key value as a data record in the source database: During full data synchronization, DTS does not synchronize the data record to the destination database. The existing data record in the destination database is retained. During incremental data synchronization, DTS synchronizes the data record to the destination database. The existing data record in the destination database is overwritten.
If the source and destination databases have different schemas, data may fail to be initialized. In this case, only some columns are synchronized, or the data synchronization instance fails. Proceed with caution.
Indexes, partitions, views, procedures, functions, triggers, and foreign keys cannot be synchronized. You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. Synchronization of INDEX, PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, and FK is not supported. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Due to the limits of AnalyticDB for MySQL, if the disk space usage of a node in the AnalyticDB for MySQL cluster exceeds 80%, the DTS task becomes abnormal and latency occurs. Estimate the required space based on the objects to be synchronized in advance to ensure that the destination cluster has sufficient storage space. If the destination AnalyticDB for MySQL V3.0 cluster is being backed up while the DTS task is running, the DTS task fails. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If a DDL operation fails to be written to the destination database, the DTS task continues to run. You need to check the task logs for the failed DDL operation. For more information about how to view task logs, see View task logs. If a DTS task fails to run, DTS technical support will try to restore the task within 8 hours. During the restoration, the task may be restarted, and the parameters of the task may be modified.
Note Only the parameters of the DTS task may be modified. The parameters of databases are not modified. The parameters that may be modified include but are not limited to the parameters in the "Modify instance parameters" section of the Modify the parameters of a DTS instance topic.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to AnalyticDB for PostgreSQL
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During synchronization, do not perform DDL operations to modify primary keys or add comments to tables, such as ALTER TABLE table_name COMMENT='Table comment';. Otherwise, the DDL operations fail to be executed during data synchronization. If the data to be synchronized in the source database contains date data of 0000-00-00 00:00:00, the task may fail.
Note DTS converts this date data to null when synchronizing it to the destination database. You can temporarily change the source data to 0001-01-01 00:00:00 or set the corresponding field in the destination database to allow null values. During schema synchronization and full data synchronization, do not execute DDL statements to change the schemas of databases or tables. Otherwise, the data synchronization task fails.
|
Other limits | Synchronization object requirements: Only table-level synchronization is supported. Synchronization of data of the BIT, VARBIT, GEOMETRY, ARRAY, UUID, TSQUERY, TSVECTOR, and TXID_SNAPSHOT types is not supported. Synchronization of prefix indexes is not supported. If the source database has prefix indexes, the data synchronization may fail. You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Synchronization of PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, FK, and INDEX is not supported.
Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If online DDL operations that use temporary tables, such as merging multiple tables, are performed on the source database, data may be lost in the destination database or the synchronization instance may fail. If a primary key or unique key conflict occurs while the synchronization instance is running: If the source and destination databases have the same schema and a data record in the destination database has the same primary key value or unique key value as a data record in the source database: During full data synchronization, DTS does not synchronize the data record to the destination database. The existing data record in the destination database is retained. During incremental data synchronization, DTS synchronizes the data record to the destination database. The existing data record in the destination database is overwritten.
If the source and destination databases have different schemas, data may fail to be initialized. In this case, only some columns are synchronized, or the data synchronization instance fails. Proceed with caution.
If the table to be synchronized has a primary key, the primary key column of the destination table must be the same as that of the source table. If the table to be synchronized does not have a primary key, the primary key column of the destination table must be the same as the distribution key. The unique key (including primary key column) of the destination table must contain all columns of a distribution key. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. AO tables are not supported as destination tables. If you use column mapping for non-full table synchronization or if the source and destination table schemas are inconsistent, data in the columns that are missing in the destination table will be lost. If a DTS task fails to run, DTS technical support will try to restore the task within 8 hours. During the restoration, the task may be restarted, and the parameters of the task may be modified.
Note Only the parameters of the DTS task may be modified. The parameters of databases are not modified. The parameters that may be modified include but are not limited to the parameters in the "Modify instance parameters" section of the Modify the parameters of a DTS instance topic.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to Alibaba Cloud DataHub
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not execute DDL statements to change the schemas of databases or tables. Otherwise, the data synchronization task fails.
|
Other limits | The maximum length of a single String field in the destination DataHub topic is 2 MB. Database-level and table-level data synchronization are supported. You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If a DTS task fails to run, DTS technical support will try to restore the task within 8 hours. During the restoration, the task may be restarted, and the parameters of the task may be modified.
Note Only the parameters of the DTS task may be modified. The parameters of databases are not modified. The parameters that may be modified include but are not limited to the parameters in the "Modify instance parameters" section of the Modify the parameters of a DTS instance topic.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to Elasticsearch
The following table describes the notes and limits.
Limit type | Description |
Limits on the source database | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not execute DDL statements to change the schemas of databases or tables. Otherwise, the data synchronization task fails.
|
Other limits | You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. To add columns to a table that you want to synchronize from the source database, perform the following steps: Modify the mappings of the table in the destination Elasticsearch cluster, execute DDL statements in the source PolarDB for MySQL cluster, and then pause and start the data synchronization task. The data types supported by PolarDB for MySQL clusters and those supported by Elasticsearch clusters do not have one-to-one correspondence. Therefore, during initial schema synchronization, DTS converts the data types of the source database to those of the destination database. For more information, see Data type mappings for schema synchronization. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases.
|
Special cases | DTS executes the CREATE DATABASE IF NOT EXISTS `test` statement in the source database as scheduled to move forward the binary log file position. |
Synchronization from PolarDB for MySQL to Alibaba Cloud Message Queue for Apache Kafka or self-managed Kafka
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not execute DDL statements to change the schemas of databases or tables. Otherwise, the data synchronization task fails.
|
Other limits | You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Synchronization of INDEX, PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, and FK is not supported. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. During synchronization, if the destination Kafka cluster is scaled out or scaled in, you must restart the instance. If a DTS task fails to run, DTS technical support will try to restore the task within 8 hours. During the restoration, the task may be restarted, and the parameters of the task may be modified.
Note Only the parameters of the DTS task may be modified. The parameters of databases are not modified. The parameters that may be modified include but are not limited to the parameters in the "Modify instance parameters" section of the Modify the parameters of a DTS instance topic.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to MaxCompute
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not execute DDL statements to change the schemas of databases or tables. Otherwise, the data synchronization task fails.
|
Other limits | You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Synchronization of INDEX, PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, and FK is not supported. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. Because MaxCompute does not support primary key constraints, if DTS retransmits data due to network issues or other reasons, duplicate records may appear in MaxCompute. If a DTS task fails to run, DTS technical support will try to restore the task within 8 hours. During the restoration, the task may be restarted, and the parameters of the task may be modified.
Note Only the parameters of the DTS task may be modified. The parameters of databases are not modified. The parameters that may be modified include but are not limited to the parameters in the "Modify instance parameters" section of the Modify the parameters of a DTS instance topic.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to self-managed Oracle
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not execute DDL statements to change the schemas of databases or tables. Otherwise, the data synchronization task fails.
|
Other limits | You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If online DDL operations that use temporary tables, such as merging multiple tables, are performed on the source database, data may be lost in the destination database or the synchronization instance may fail. If a primary key or unique key conflict occurs while the synchronization instance is running: If the source and destination databases have the same schema and a data record in the destination database has the same primary key value or unique key value as a data record in the source database: During full data synchronization, DTS does not synchronize the data record to the destination database. The existing data record in the destination database is retained. During incremental data synchronization, DTS synchronizes the data record to the destination database. The existing data record in the destination database is overwritten.
If the source and destination databases have different schemas, data may fail to be initialized. In this case, only some columns are synchronized, or the data synchronization instance fails. Proceed with caution.
While the DTS instance is running, disable the enabled triggers and foreign keys in the destination database. Otherwise, the DTS instance fails. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If the self-managed Oracle database uses a Real Application Clusters (RAC) architecture, you cannot configure a Single Client Access Name (SCAN) IP address. You can only configure one of the virtual IP addresses (VIPs) in the connection information. After this configuration, node switching in the RAC is not supported. If a DTS task fails to run, DTS technical support will try to restore the task within 8 hours. During the restoration, the task may be restarted, and the parameters of the task may be modified.
Note Only the parameters of the DTS task may be modified. The parameters of databases are not modified. The parameters that may be modified include but are not limited to the parameters in the "Modify instance parameters" section of the Modify the parameters of a DTS instance topic.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |