Precautions and limits for synchronizing data from a PolarDB-X 1.0 instance - Data Transmission Service

This topic describes the precautions and limits when you synchronize data from a PolarDB-X 1.0 instance. To ensure that your data synchronization task runs as expected, you must read the precautions and limits before you configure the task.

Important

For more information about the limits for using PolarDB-X 1.0 as the source instance, see Limits for using PolarDB-X as the source instance.

For more information about the solutions for using PolarDB-X 1.0 as the source instance, see Solutions for using DRDS as the source instance.

Scenarios of synchronizing data from a PolarDB-X 1.0 instance

Take note of precautions and limits in the following data synchronization scenarios:

Note

By default, Data Transmission Service (DTS) disables foreign key constraints for the destination database in a data synchronization task. Therefore, the cascade and delete operations of the source database are not synchronized to the following types of destination databases:

PolarDB-X 1.0
ApsaraDB RDS for MySQL instance or self-managed MySQL database
PolarDB MySQL
AnalyticDB for MySQL cluster

Synchronize data between PolarDB-X 1.0 instances
Synchronize data from a PolarDB-X 1.0 instance to a MySQL database or PolarDB for MySQL cluster
Synchronize data from a PolarDB-X 1.0 instance to an AnalyticDB for MySQL cluster
Synchronize data from a PolarDB-X 1.0 instance to a DataHub project

Synchronize data between PolarDB-X 1.0 instances

Category	Migration description
Limits on the source database	The tables to be synchronized must have PRIMARY KEY or UNIQUE constraints and all fields must be unique. Otherwise, the destination database may contain duplicate data records. The tables that only have UNIQUE constraints do not support schema synchronization. Therefore, we recommend that you synchronize the tables that have PRIMARY KEY constraints. Tables that have secondary indexes cannot be synchronized. If you select tables as the objects to be synchronized and you want to edit the tables (such as renaming tables or columns) in the destination database, you can synchronize up to 5,000 tables in a single data synchronization task. If you run a task to synchronize more than 5,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database. The following requirements for binary logs of the ApsaraDB RDS for MySQL instances attached to the PolarDB-X 1.0 instance must be met: The binary logging feature is enabled. The value of the binlog_row_image parameter is set to full. Otherwise, error messages are returned during the precheck and the data synchronization task cannot be started. For an incremental data synchronization task, the binary logs of the source database must be retained for at least 24 hours. For a full and incremental data synchronization task, the binary logs of the source database must be retained for at least seven days. Otherwise, DTS may fail to obtain the binary logs and the task may fail. In exceptional circumstances, data may be inconsistent or lost. After full data synchronization is complete, you can set the retention period to more than 24 hours. Make sure that you set the retention period of binary logs in accordance with the preceding requirements. Otherwise, the service reliability and performance stated in the Service Level Agreement (SLA) of DTS cannot be achieved. Limits on operations to be performed on the source database: If you change the network type of the PolarDB-X 1.0 instance during data synchronization, you must modify the network connection information of the data synchronization task after you change the network type. During data synchronization, do not scale the source instance, including the ApsaraDB RDS for MySQL instance attached to the source instance, or change the distribution of physical databases for which a logical database or logical table has been configured in the ApsaraDB RDS for MySQL instance. In addition, do not migrate hot tables, change the shard keys, or perform online DDL operations on the source instance. Otherwise, the data synchronization task fails or data inconsistency occurs.ApsaraDB RDS for MySQL
Other limits	The storage type of the PolarDB-X 1.0 instance must be ApsaraDB RDS for MySQL, including custom instances and purchased instances. PolarDB for MySQL cannot be used as the storage type. Read-only instances at the PolarDB-X 1.0 compute layers are not supported. PolarDB-X 1.0 storage resources can be split only by using horizontal splitting. Both databases and tables can be split. Vertical splitting is not supported. When DTS synchronizes data from a PolarDB-X 1.0 instance, data is distributed across the attached ApsaraDB RDS for MySQL instances. DTS runs a subtask for each ApsaraDB RDS for MySQL instance. The running state of the subtask is displayed in Task Topology. Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads on the database servers. During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After full data synchronization is complete, the size of used tablespace of the destination database is larger than that of the source database. We recommend that you do not use gh-ost or pt-online-schema-change to perform DDL operations on objects to be synchronized. Otherwise, data synchronization may fail. During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. If you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

Synchronize data from a PolarDB-X 1.0 instance to a MySQL database or PolarDB for MySQL cluster

Category	Migration description
Limits on the source database	The tables to be synchronized must have PRIMARY KEY or UNIQUE constraints and all fields must be unique. Otherwise, the destination database may contain duplicate data records. The tables that only have UNIQUE constraints do not support schema synchronization. Therefore, we recommend that you synchronize the tables that have PRIMARY KEY constraints. Tables that have secondary indexes cannot be synchronized. If you select tables as the objects to be synchronized and you want to edit the tables (such as renaming tables or columns) in the destination database, you can synchronize up to 5,000 tables in a single data synchronization task. If you run a task to synchronize more than 5,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database. The following requirements for binary logs of the ApsaraDB RDS for MySQL instances attached to the PolarDB-X 1.0 instance must be met: The binary logging feature is enabled. The value of the binlog_row_image parameter is set to full. Otherwise, error messages are returned during the precheck and the data synchronization task cannot be started. For an incremental data synchronization task, the binary logs of the source database must be retained for at least 24 hours. For a full and incremental data synchronization task, the binary logs of the source database must be retained for at least seven days. Otherwise, DTS may fail to obtain the binary logs and the task may fail. In exceptional circumstances, data may be inconsistent or lost. After full data synchronization is complete, you can set the retention period to more than 24 hours. Make sure that you set the retention period of binary logs in accordance with the preceding requirements. Otherwise, the service reliability and performance stated in the Service Level Agreement (SLA) of DTS cannot be achieved. Limits on operations to be performed on the source database: If you change the network type of the PolarDB-X 1.0 instance during data synchronization, you must modify the network connection information of the data synchronization task after you change the network type. During data synchronization, do not scale the source instance, including the ApsaraDB RDS for MySQL instance attached to the source instance, or change the distribution of physical databases for which a logical database or logical table has been configured in the ApsaraDB RDS for MySQL instance. In addition, do not migrate hot tables, change the shard keys, or perform online DDL operations on the source instance. Otherwise, the data synchronization task fails or data inconsistency occurs.ApsaraDB RDS for MySQL The storage type of the PolarDB-X 1.0 instance must be ApsaraDB RDS for MySQL, including custom instances and purchased instances. PolarDB for MySQL cannot be used as the storage type. PolarDB-X 1.0 storage resources can be split only by using horizontal splitting. Both databases and tables can be split. Vertical splitting is not supported. Read-only instances at the PolarDB-X 1.0 compute layers are not supported.
Other limits	When DTS synchronizes data from a PolarDB-X 1.0 instance, data is distributed across the attached ApsaraDB RDS for MySQL instances. DTS runs a subtask for each ApsaraDB RDS for MySQL instance. The running state of the subtask is displayed in Task Topology. Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads on the database servers. During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After full data synchronization is complete, the size of used tablespace of the destination database is larger than that of the source database. We recommend that you do not use gh-ost or pt-online-schema-change to perform DDL operations on objects to be synchronized. Otherwise, data synchronization may fail. During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. If you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

Synchronize data from a PolarDB-X 1.0 instance to an AnalyticDB for MySQL cluster

Category	Migration description
Limits on the source database	The tables to be synchronized must have PRIMARY KEY or UNIQUE constraints and all fields must be unique. Otherwise, the destination database may contain duplicate data records. The tables that only have UNIQUE constraints do not support schema synchronization. Therefore, we recommend that you synchronize the tables that have PRIMARY KEY constraints. Tables that have secondary indexes cannot be synchronized. If you select tables as the objects to be synchronized and you want to edit the tables (such as renaming tables or columns) in the destination database, you can synchronize up to 5,000 tables in a single data synchronization task. If you run a task to synchronize more than 5,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database. The following requirements for binary logs of the ApsaraDB RDS for MySQL instances attached to the PolarDB-X 1.0 instance must be met: The binary logging feature is enabled. The value of the binlog_row_image parameter is set to full. Otherwise, error messages are returned during the precheck and the data synchronization task cannot be started. For an incremental data synchronization task, the binary logs of the source database must be retained for at least 24 hours. For a full and incremental data synchronization task, the binary logs of the source database must be retained for at least seven days. Otherwise, DTS may fail to obtain the binary logs and the task may fail. In exceptional circumstances, data may be inconsistent or lost. After full data synchronization is complete, you can set the retention period to more than 24 hours. Make sure that you set the retention period of binary logs in accordance with the preceding requirements. Otherwise, the service reliability and performance stated in the Service Level Agreement (SLA) of DTS cannot be achieved. Limits on operations to be performed on the source database: If you change the network type of the PolarDB-X 1.0 instance during data synchronization, you must modify the network connection information of the data synchronization task after you change the network type. During data synchronization, do not scale the source instance, including the ApsaraDB RDS for MySQL instance attached to the source instance, or change the distribution of physical databases for which a logical database or logical table has been configured in the ApsaraDB RDS for MySQL instance. In addition, do not migrate hot tables, change the shard keys, or perform online DDL operations on the source instance. Otherwise, the data synchronization task fails or data inconsistency occurs.ApsaraDB RDS for MySQL The storage type of the PolarDB-X 1.0 instance must be ApsaraDB RDS for MySQL, including custom instances and purchased instances. PolarDB for MySQL cannot be used as the storage type. PolarDB-X 1.0 storage resources can be split only by using horizontal splitting. Both databases and tables can be split. Vertical splitting is not supported. Read-only instances at the PolarDB-X 1.0 compute layers are not supported.
Other limits	When DTS synchronizes data from a PolarDB-X 1.0 instance, data is distributed across the attached ApsaraDB RDS for MySQL instances. DTS runs a subtask for each ApsaraDB RDS for MySQL instance. The running state of the subtask is displayed in Task Topology. Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads on the database servers. During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After full data synchronization is complete, the size of used tablespace of the destination database is larger than that of the source database. We recommend that you do not use gh-ost or pt-online-schema-change to perform DDL operations on objects to be synchronized. Otherwise, data synchronization may fail. During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. If you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.

Synchronize data from a PolarDB-X 1.0 instance to a DataHub project

Category	Migration description
Limits on the source database	The tables to be synchronized must have PRIMARY KEY or UNIQUE constraints and all fields must be unique. Otherwise, the destination database may contain duplicate data records. The tables that only have UNIQUE constraints do not support schema synchronization. Therefore, we recommend that you synchronize the tables that have PRIMARY KEY constraints. Tables that have secondary indexes cannot be synchronized. If you select tables as the objects to be synchronized and you want to edit the tables (such as renaming tables or columns) in the destination database, you can synchronize up to 5,000 tables in a single data synchronization task. If you run a task to synchronize more than 5,000 tables, a request error occurs. In this case, we recommend that you configure multiple tasks to synchronize the tables in batches or configure a task to synchronize the entire database. The following requirements for binary logs of the ApsaraDB RDS for MySQL instances attached to the PolarDB-X 1.0 instance must be met: The binary logging feature is enabled. The value of the binlog_row_image parameter is set to full. Otherwise, error messages are returned during the precheck and the data synchronization task cannot be started. For an incremental data synchronization task, the binary logs of the source database must be retained for at least 24 hours. For a full and incremental data synchronization task, the binary logs of the source database must be retained for at least seven days. Otherwise, DTS may fail to obtain the binary logs and the task may fail. In exceptional circumstances, data may be inconsistent or lost. After full data synchronization is complete, you can set the retention period to more than 24 hours. Make sure that you set the retention period of binary logs in accordance with the preceding requirements. Otherwise, the service reliability and performance stated in the Service Level Agreement (SLA) of DTS cannot be achieved. Limits on operations to be performed on the source database: If you change the network type of the PolarDB-X 1.0 instance during data synchronization, you must modify the network connection information of the data synchronization task after you change the network type. During data synchronization, do not scale the source instance, including the ApsaraDB RDS for MySQL instance attached to the source instance, or change the distribution of physical databases for which a logical database or logical table has been configured in the ApsaraDB RDS for MySQL instance. In addition, do not migrate hot tables, change the shard keys, or perform online DDL operations on the source instance. Otherwise, the data synchronization task fails or data inconsistency occurs.ApsaraDB RDS for MySQL The storage type of the PolarDB-X 1.0 instance must be ApsaraDB RDS for MySQL, including custom instances and purchased instances. PolarDB for MySQL cannot be used as the storage type. PolarDB-X 1.0 storage resources can be split only by using horizontal splitting. Both databases and tables can be split. Vertical splitting is not supported. Read-only instances at the PolarDB-X 1.0 compute layers are not supported.
Other limits	Only incremental data synchronization and schema synchronization are supported. Full data synchronization is not supported. Limits on the objects to be synchronized: Only tables can be selected as the objects to be synchronized. After a data synchronization task is started, DTS does not synchronize columns that are created in the source database to the destination DataHub project. When DTS synchronizes data from a PolarDB-X 1.0 instance, data is distributed across the attached ApsaraDB RDS for MySQL instances. DTS runs a subtask for each ApsaraDB RDS for MySQL instance. The running state of the subtask is displayed in Task Topology. Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses the read and write resources of the source and destination databases. This may increase the loads on the database servers. During initial full data synchronization, concurrent INSERT operations cause fragmentation in the tables of the destination database. After full data synchronization is complete, the size of used tablespace of the destination database is larger than that of the source database. We recommend that you do not use gh-ost or pt-online-schema-change to perform DDL operations on objects to be synchronized. Otherwise, data synchronization may fail. During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases. If you use tools other than DTS to write data to the destination database, data loss may occur in the destination database when you use DMS to perform online DDL operations.