Synchronize RDS for MySQL to AnalyticDB for MySQL 3.0 - Data Transmission Service

This topic describes how to synchronize data from an ApsaraDB RDS for MySQL instance to an AnalyticDB for MySQL V3.0 cluster by using Data Transmission Service (DTS). After you synchronize data, you can use AnalyticDB for MySQL to build internal business intelligence (BI) systems, interactive query systems, and real-time report systems.

Prerequisites

An ApsaraDB RDS for MySQL instance is created. For more information, see Create an ApsaraDB RDS for MySQL instance.
The destination AnalyticDB for MySQL V3.0 cluster is created. For more information, see Create a cluster.
The available storage space of the destination AnalyticDB for MySQL V3.0 cluster is larger than the total size of the data in the source ApsaraDB RDS for MySQL instance.

Usage notes

Note

During schema synchronization, DTS does not synchronize foreign keys from the source database to the destination database.
During full data synchronization and incremental data synchronization, DTS temporarily disables constraint checks and foreign key cascade operations at the session level. If you perform cascade update or delete operations on the source database while the task is running, data inconsistency may occur.
During schema synchronization, DTS does not support synchronizing materialized views to the target AnalyticDB for MySQL instance. If you need to use materialized views, manually create them in the target instance after the synchronization is complete.

Type	Description
Source database limits	The tables to synchronize must have a primary key or a unique constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize at the table level and need to edit mappings (such as column name mapping), each synchronization task supports up to 1,000 tables. If you exceed this limit, the task fails with an error. To fix this, split the tables across multiple tasks or configure a full-database synchronization task. Binary logs: ApsaraDB RDS for MySQL enables binary logging by default. Ensure that the binlog_row_image parameter is set to full. Otherwise, the precheck fails and the synchronization task cannot start. For instructions, see Configure instance parameters. Important If your source instance is a self-managed MySQL database, enable binary logging and set binlog_format to row and binlog_row_image to full. If your self-managed MySQL database is a dual-primary cluster (where both nodes act as primary and secondary), enable the log_slave_updates parameter so DTS can capture all binary log events. For instructions, see Create an account and configure binary logging for a self-managed MySQL database. The local binary logs for an ApsaraDB RDS for MySQL instance must be retained for at least three days (seven days is recommended). For a self-managed MySQL database, retain local binary logs for at least seven days. Otherwise, DTS may fail to retrieve binary logs, causing the task to fail. In extreme cases, this may cause data inconsistency or data loss. Issues caused by binary log retention periods shorter than DTS requires are not covered under the DTS SLA. Note To configure the retention period for local binary logs on an ApsaraDB RDS for MySQL instance, see Automatically delete local logs. Do not run DDL operations that modify primary keys or add comments—such as `ALTER TABLE table_name COMMENT='Table comment';`—while synchronization is running. Otherwise, the DDL operation fails. Do not run DDL operations that change database or table schemas during schema synchronization or full synchronization. Otherwise, the synchronization task fails. Note During full synchronization, DTS queries the source database. This creates metadata locks that may block DDL operations on the source database. Data generated by changes that do not write to binary logs—such as data restored from physical backups or created by cascade operations—is not synchronized to the destination database. Note If this occurs, remove the affected database or table from the synchronization objects. Then add it back. You can do this only if your business allows it. For more information, see Modify synchronization objects. If your source database is MySQL 8.0.23 or later and contains invisible hidden columns, DTS may not read those columns. This may cause data loss. Note Run the `ALTER TABLE <table_name> ALTER COLUMN <column_name> SET VISIBLE;` command to make the hidden column visible. For more information, see Invisible Columns.
Other limits	DTS does not support synchronizing prefix indexes. If your source database contains prefix indexes, synchronization may fail. If your source database uses online DDL operations in temporary table mode—including but not limited to multi-table merge scenarios—or adds function-based indexes to unique key columns, data loss or task failure may occur in the destination database. DTS does not support synchronizing indexes, partitions, views, stored procedures, functions, triggers, or foreign keys. If a primary key or unique key conflict occurs while the task is running: If the table schemas are consistent and a record in the destination database has the same primary key or unique key value as a record in the source database: During full data synchronization, DTS retains the destination record and skips the source record. During incremental synchronization, DTS overwrites the destination record with the source record. If the table schemas are inconsistent, data initialization may fail. This can result in only partial data synchronization or a complete synchronization failure. Use with caution. The destination database must contain a custom primary key. Or, in the Configurations for Databases, Tables, and Columns step, configure a Primary Key Column. Otherwise, synchronization may fail. Due to usage limits in AnalyticDB for MySQL, if disk usage exceeds 80% in an AnalyticDB for MySQL cluster, DTS tasks may experience delays or errors. Estimate required space based on your synchronization objects and ensure sufficient storage in the destination cluster. If the destination AnalyticDB for MySQL 3.0 cluster is backing up while the DTS task runs, the task fails. Assess the performance of both the source and destination databases before starting synchronization. Run synchronization during off-peak hours. Otherwise, full initialization consumes read and write resources on both databases and may increase database load. Full initialization runs INSERT operations concurrently. This fragments destination tables. After full initialization, the tablespace of the destination instance is larger than that of the source instance. If you synchronize one or more tables—not a full database—do not use tools like pt-online-schema-change to perform online DDL operations on the source tables. Otherwise, synchronization fails. You can use Data Management (DMS) to perform online DDL operations. For more information, see Online DDL without locking tables. For table-level synchronization, if no data other than DTS writes to the destination AnalyticDB for MySQL, you can use Data Management (DMS) to perform online DDL operations. For more information, see Online DDL without locking tables. Do not write data to the destination database except through DTS while synchronization is running. Otherwise, data inconsistency may occur between the source and destination databases. For example, if you use DMS to perform online DDL operations while other data is written to the destination database, data loss may occur. If a DDL operation fails in the destination database, the DTS task continues to run. Check the failed DDL in the task logs. For instructions, see View task logs. If your ApsaraDB RDS for MySQL instance has Always-Encrypted enabled, full data synchronization is not supported. Note ApsaraDB RDS for MySQL instances with Transparent Data Encryption (TDE) enabled support schema synchronization, full data synchronization, and incremental data synchronization. If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters. Note Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters.
Special cases	For a self-managed MySQL source database: If a primary/secondary switchover occurs in the source database during synchronization, the task fails. DTS calculates latency by comparing the timestamp of the last synchronized record with the current time. If no DML operations run for a long time in the source database, latency reporting may become inaccurate. If latency appears too high, run a DML operation in the source database to update the latency. Note If you select a full database for synchronization, create a heartbeat table. Update or write to this table every second. DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command in the source database to advance the binary log offset. If your source database is Amazon Aurora MySQL or another clustered MySQL instance, ensure the domain name or IP address used in the task configuration—and its DNS resolution—always points to a read/write (RW) node. Otherwise, synchronization may fail. For an ApsaraDB RDS for MySQL source database: Read-only instances—such as ApsaraDB RDS for MySQL 5.6 read-only instances—that do not record transaction logs cannot serve as source databases. DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command in the source database to advance the binary log offset.

Billing

Synchronization type	Pricing
Schema synchronization and full data synchronization	Free of charge.
Incremental data synchronization	Charged. For more information, see Billing overview.

Supported synchronization topologies

One-way one-to-one synchronization
One-way one-to-many synchronization
One-way many-to-one synchronization

SQL operations that can be synchronized

Operation type	SQL statement
DML	INSERT, UPDATE, and DELETE Note When data is written to the AnalyticDB for MySQL cluster, the UPDATE statement is automatically converted to the REPLACE INTO statement.
DDL	CREATE TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE, ADD COLUMN, and DROP COLUMN

Warning

If the data type of a field in the source table is changed during data synchronization, an error message is returned and the data synchronization task is interrupted. You can perform the following steps to troubleshoot the issue:

When DTS synchronizes data to the destination AnalyticDB for MySQL cluster, the data type of a field in the source table is changed. As a result, the data synchronization task fails. In this example, the source table is named customer.
In the destination AnalyticDB for MySQL V3.0 cluster, create a table named customer_new that has the same schema as the customer table.
Execute the INSERT INTO SELECT statement to copy the data of the customer table and insert the data into the customer_new table. This ensures that the data of the two tables is consistent.
Rename or delete the customer table. Then, change the name of the customer_new table to customer.
Start the data synchronization task again in the DTS console.

Procedure

Go to the Data Synchronization Tasks page.
1. Log on to the Data Management (DMS) console.
2. In the top navigation bar, click Data + AI.
3. In the left-side navigation pane, choose DTS (DTS) > Data Synchronization.
Note
- Operations may vary based on the mode and layout of the DMS console. For more information, see Simple mode console and Customize DMS console layout and style.
- You can also go to the Data Synchronization Tasks page of the new DTS console.
On the right side of Data Synchronization Tasks, select the region in which the data synchronization instance resides.

Note
If you use the new DTS console, you must select the region in which the data synchronization instance resides in the top navigation bar.

Click Create Task to configure the source and destination databases.

Warning

After you select the source and destination instances, review the Limits at the top of the page. Otherwise, the task may fail or data inconsistency may occur.

Section	Parameter	Description
N/A	Task Name	DTS automatically generates a task name. We recommend that you specify a descriptive name for easy identification. The name does not need to be unique.
Source Database	Select DMS Database Instance	Choose whether to use an existing instance as needed. If you use an existing instance, the database information is automatically filled in. You do not need to enter it again. If you do not use an existing instance, enter the database information below.
	Database Type	The type of the source database. Select MySQL.
	Connection Type	The access method of the source database. Select Alibaba Cloud Instance.
	Instance Region	The region in which the source ApsaraDB RDS for MySQL instance resides.
	Cross-account	Specifies whether data is synchronized across Alibaba Cloud accounts. In this example, No is selected.
	RDS Instance ID	The ID of the source ApsaraDB RDS for MySQL instance.
	Database Account	The database account of the source ApsaraDB RDS for MySQL instance. The account must have the SELECT permissions on the objects to be synchronized and the REPLICATION CLIENT and REPLICATION SLAVE permissions.
	Database Password	Enter the password for the specified database account.
	Connection Method	Select Non-encrypted or SSL-encrypted as needed. If you set this to SSL-encrypted, you must enable SSL encryption for the RDS for MySQL instance beforehand. For more information, see Use a cloud certificate to quickly enable SSL link encryption.
Destination Database	Select DMS Database Instance	Choose whether to use an existing instance as needed. If you use an existing instance, the database information is automatically filled in. You do not need to enter it again. If you do not use an existing instance, enter the database information below.
	Database Type	The type of the destination database. Select AnalyticDB for MySQL 3.0.
	Connection Type	The access method of the destination database. Select Alibaba Cloud Instance.
	Instance Region	The region in which the destination AnalyticDB for MySQL V3.0 cluster resides.
	Instance ID	The ID of the destination AnalyticDB for MySQL V3.0 cluster.
	Database Account	The database account of the destination AnalyticDB for MySQL V3.0 cluster. The account must have read and write permissions on the destination database.
	Database Password	Enter the password for the specified database account.

After completing the configuration, click Test Connection and Proceed at the bottom of the page.
If the source or destination database is an Alibaba Cloud database instance, such as an RDS for MySQL instance or an ApsaraDB for MongoDB instance, DTS automatically adds the IP address of the DTS service for the corresponding region to the whitelist of the Alibaba Cloud database instance. If the source or destination database is a self-managed database on an ECS instance, DTS automatically adds the IP address of the DTS service for the corresponding region to the security rules of the ECS instance. You must also ensure that the self-managed database does not restrict access from the ECS instance. If the database is deployed in a cluster on multiple ECS instances, you must manually add the IP address of the DTS service for the corresponding region to the security rules of each of the other ECS instances. If the source or destination database is a self-managed database in an IDC or a database from another cloud provider, you must manually add the IP address of the DTS service for the corresponding region to allow access from the DTS servers. For more information about the IP addresses of the DTS service, see IP address blocks of DTS servers.
Warning
Adding the public IP address blocks of the DTS service, either automatically or manually, may pose security risks. Using this product, you acknowledge that you understand and accept the potential security risks and that you must implement basic security measures. These measures include, but are not limited to, strengthening password security, limiting the ports open to each CIDR block, using authentication for internal API calls, and regularly checking and restricting unnecessary CIDR blocks. Alternatively, you can connect through a private network using a leased line, VPN Gateway, or Smart Access Gateway.

Configure the task objects and advanced settings.

Basic Settings

Parameter or setting	Description
Synchronization Type	DTS always selects Incremental Data Synchronization. By default, you must also select Schema Synchronization and Full Data Synchronization. After the precheck, DTS initializes the destination cluster with the full data of the selected source objects, which serves as the baseline for subsequent incremental synchronization. Note If Full Data Synchronization is selected, you can synchronize the schema and data of the table that is created by executing the CREATE TABLE statement to the destination database.
Processing Mode for Existing Destination Tables	Precheck and Report Errors: Checks for tables with the same names in the destination database. If any tables with the same names are found, an error is reported during the precheck and the data synchronization task does not start. Otherwise, the precheck is successful. Note If you cannot delete or rename the table with the same name in the destination database, you can map it to a different name in the destination. For more information, see Object name mapping. Ignore Errors and Proceed: Skips the check for tables with the same name in the destination database. Warning Selecting Ignore Errors and Proceed may cause data inconsistency and put your business at risk. For example: If the table schemas are consistent and a record in the destination database has the same primary key or unique key value as a record in the source database: During full data synchronization, DTS retains the destination record and skips the source record. During incremental synchronization, DTS overwrites the destination record with the source record. If the table schemas are inconsistent, data initialization may fail. This can result in only partial data synchronization or a complete synchronization failure. Use with caution.
Select DDL and DML for Instance-Level Synchronization	The DDL and DML operations that you want to synchronize. For more information, see SQL operations that can be synchronized. Note To select the SQL operations performed on a specific database or table, right-click an object in the Selected Objects section. In the dialog box that appears, select the SQL operations that you want to synchronize.
Table Merging	Select Yes: In online transactional processing (TP) scenarios, sharding is often used to improve the response speed of business tables. In online analytical processing (OLAP) scenarios, a single table in the destination database can store large volumes of data, which simplifies single-table queries. In such scenarios, you can use the DTS table merging feature to synchronize multiple tables with the same schema (sharded tables) from the source database to a single table in the destination database. For details on this operation, see Enable table merging. Note After you select multiple tables from the source database, you must use the object name mapping feature to change their names to the same table name in the destination database. For more information about the object name mapping feature, see Object name mapping. DTS adds a `__dts_data_source` column of the TEXT type to the destination table to store the data source. The column value is written in the `DTS instance ID:database name:schema name:table name` format to distinguish the source of the table, for example, `dts******:dtstestdata:testschema:customer1`. The table merging feature applies at the task level, not the table level. To merge only some tables, you must create a separate data synchronization task for them. Warning** Do not perform DDL operations that change the database or table schema in the source database. Otherwise, data inconsistency or task failure may occur. No: This is the default option.
Case Policy for Destination Object Names	Configure the case-sensitivity policy for database, table, and column names in the destination instance. By default, the DTS default policy is selected. You can also choose to use the default policy of the source or destination database. For more information, see Case policy for destination object names.
Source Objects	In the Source Objects box, click the objects, and then click to move them to the Selected Objects box. Note You can select columns, tables, or databases as the objects to be synchronized. If you select tables or columns as the objects to be synchronized, DTS does not synchronize other objects such as views, triggers, or stored procedures to the destination database. If you select a database as the object to be synchronized, DTS synchronizes data based on the following default rules: If the table to be synchronized in the source database has a primary key, such as a single-column primary key or a composite primary key, the primary key columns are specified as the distribution keys. If the table to be synchronized in the source database does not have a primary key, an auto-increment primary key column is automatically generated in the destination table. This may cause data inconsistency between the source and destination databases.
Selected Objects	To rename a single object in the destination instance, right-click the object in the Selected Objects box. For more information, see Map a single object name. To rename multiple objects in bulk, click Batch Edit in the upper-right corner of the Selected Objects box. For more information, see Map multiple object names in bulk. Note To select the SQL operations performed on a specific database or table, right-click an object to be synchronized in the Selected Objects section. In the dialog box that appears, select the SQL operations that you want to synchronize. For more information about the SQL operations that can be synchronized, see SQL operations that can be synchronized. To specify WHERE conditions to filter data, right-click an object in the Selected Objects section. In the dialog box that appears, specify the conditions. For more information about how to specify the conditions, see Use SQL conditions to filter data.

Advanced Settings

Parameter	Description
Monitoring and Alerting	Choose whether to set up alerts. If the synchronization fails or the latency exceeds the specified threshold, DTS sends a notification to the alert contacts. No: No alerts are configured. Yes: Configures alerts. You must also set the alert threshold and alert notifications. For more information, see Configure monitoring and alerting during task configuration.
Replicate Online DDL Temporary Tables to the Destination Database	DDL operations cannot be synchronized. Select No, Adapt To DMS Online DDL or No, Adapt To Gh-ost. If DDL synchronization is supported for this scenario in the future, you can choose whether to synchronize the temporary tables that are generated when you use Data Management (DMS) or gh-ost for online DDL changes. Yes: Synchronizes data in temporary tables created by online DDL changes. Note If the data of temporary tables generated by online DDL changes is too large, it may cause synchronization latency. No, Adapt to DMS Online DDL: Does not synchronize data from temporary tables generated by online DDL changes. Only the original DDL statements executed in Data Management (DMS) are synchronized. Note This option causes locked tables in the destination database. Do not adapt to gh-ost: This option does not synchronize data from temporary tables generated by online DDL changes. Instead, it only synchronizes the original DDL statements executed by gh-ost. You can use default or custom regular expressions for gh-ost shadow and junk tables. Note This option causes locked tables in the destination database.
Retry Interval After Source or Destination Database Connection Failure	If the connection to the source or destination database fails after the synchronization task starts, DTS reports an error and immediately begins to retry the connection. The default retry duration is 720 minutes. You can customize the retry time to a value from 10 to 1,440 minutes. We recommend a duration of 30 minutes or more. If the connection is restored within this period, the task resumes automatically. Otherwise, the task fails. Note If multiple DTS instances (e.g., Instance A and B) share a source or destination, DTS uses the shortest configured retry duration (e.g., 30 minutes for A, 60 for B, so 30 minutes is used) for all instances. DTS charges for task runtime during connection retries. Set a custom duration based on your business needs, or release the DTS instance promptly after you release the source/destination instances.
Configure ETL	Choose whether to enable the extract, transform, and load (ETL) feature. For more information, see What is ETL? Valid values: Yes: Enables the ETL feature. Enter data processing statements in the code editor. For more information, see Configure ETL in a data migration or data synchronization task. No: Disables the ETL feature.

Optional: After you complete the configuration, click Next: Configure Database and Table Fields to set the Type, Primary Key Column, Distribution Key, and partition key information (Partition Key, Partitioning Rules, and Partition Lifecycle) for the tables to sync in the destination database.
Note
- This step is available only if you select the Schema Synchronization checkbox for Synchronization Types during task object configuration. You can set Definition Status to All to make modifications.
- You can use the Primary Key Column to specify a composite primary key that consists of multiple columns. You must then select one or more columns from the Primary Key Column to serve as the Distribution Key and Partition Key. For more information, see CREATE TABLE.
Save the task and perform a precheck.
- To view the parameters for configuring this instance via an API operation, hover over the Next: Save Task Settings and Precheck button and click Preview OpenAPI parameters in the tooltip.
- If you have finished viewing the API parameters, click Next: Save Task Settings and Precheck at the bottom of the page.
Note
- Before a synchronization task starts, DTS performs a precheck. You can start the task only if the precheck passes.
- If the precheck fails, click View Details next to the failed item, fix the issue as prompted, and then rerun the precheck.
- If the precheck generates warnings:
  For non-ignorable warning, click View Details next to the item, fix the issue as prompted, and run the precheck again.
  For ignorable warnings, you can bypass them by clicking Confirm Alert Details, then Ignore, and then OK. Finally, click Precheck Again to skip the warning and run the precheck again. Ignoring precheck warnings may lead to data inconsistencies and other business risks. Proceed with caution.
When the Success Rate reaches 100%, click Next: Purchase Instance.

On the Purchase page, select the billing method and link specifications for the data synchronization instance. For more information, see the following table.

Category	Parameter	Description
New Instance Class	Billing Method	Subscription: You pay upfront for a specific duration. This is cost-effective for long-term, continuous tasks. Pay-as-you-go: You are billed hourly for actual usage. This is ideal for short-term or test tasks, as you can release the instance at any time to save costs.
	Resource Group Settings	The resource group to which the instance belongs. The default is default resource group. For more information, see What is Resource Management?.
	Instance Class	DTS offers synchronization specifications at different performance levels that affect the synchronization rate. Select a specification based on your business requirements. For more information, see Data synchronization link specifications.
	Subscription Duration	In subscription mode, select the duration and quantity of the instance. Monthly options range from 1 to 9 months. Yearly options include 1, 2, 3, or 5 years. Note This option appears only when the billing method is Subscription.

Read and select the checkbox for Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start, and then click OK in the OK dialog box.
You can monitor the task progress on the data synchronization page.