Data Transmission Service (DTS) supports data synchronization from an ApsaraDB RDS for MySQL instance to a serverless AnalyticDB for PostgreSQL instance. You can use this feature to centralize your enterprise data for analysis.
Prerequisites
A source ApsaraDB RDS for MySQL instance is created. For more information, see Create a RDS MySQL instance.
You must have a target AnalyticDB for PostgreSQL instance in serverless mode with a kernel version of V1.0.3.1 or later.
To create an AnalyticDB for PostgreSQL instance in serverless mode, see Create an instance.
To view the minor version and upgrade the version, see View the minor version and Version upgrade.
Supported MySQL types
You can synchronize data from the following MySQL source databases to an AnalyticDB for PostgreSQL instance. This topic uses an ApsaraDB RDS for MySQL instance to demonstrate the configuration. The procedure is similar for other source databases.
-
ApsaraDB RDS for MySQL instance
-
Self-managed database that is hosted on Elastic Compute Service (ECS)
-
Self-managed database that is connected over Express Connect, VPN Gateway, or Smart Access Gateway
-
Self-managed database that is connected over Database Gateway
-
Self-managed database that is connected over Cloud Enterprise Network (CEN)
In addition to MySQL, DTS supports PostgreSQL, SQL Server, and DB2 as data sources. For more information about supported databases, see Supported databases.
Usage notes
By default, DTS disables foreign key constraints when synchronizing data to the destination database. Therefore, DTS does not synchronize cascade and delete operations from the source database.
Type | Description |
Source database limits |
|
Other limits |
|
Special cases |
|
Billing
|
Synchronization type |
Pricing |
|
Schema synchronization and full data synchronization |
Free of charge. |
|
Incremental data synchronization |
Charged. For more information, see Billing overview. |
Supported synchronization topologies
-
One-to-one, one-way synchronization.
-
One-to-many, one-way synchronization.
-
Many-to-one, one-way synchronization.
Supported SQL operations
-
DML operations: INSERT, UPDATE, and DELETE.
-
DDL operation: ADD COLUMN.
NoteThe CREATE TABLE operation is not supported. To synchronize a new table, you must add it as a synchronization object.
Term mappings
|
MySQL |
AnalyticDB for PostgreSQL |
|
Database |
Schema |
|
Table |
Table |
Procedure
Log on to the Data Synchronization Tasks page of the new DTS console.
NoteYou can also log on to the Data Management Service (DMS) console. In the top menu bar, choose Integration and Development (DTS). In the left-side navigation pane, choose .
In the upper-left corner, select the region of the data synchronization instance.
Click Create Task to configure the source and destination databases.
Category
Parameter
Description
N/A
Job Name
DTS automatically generates a task name. We recommend using a descriptive name for easy identification. The name does not need to be unique.
Source Database
Select Instance
Select an existing ApsaraDB RDS for MySQL instance. This parameter is optional.
Database Engine
Select MySQL.
Access Method
Select Alibaba Cloud Instance.
Instance Region
Select the region where the source ApsaraDB RDS for MySQL instance is located.
Replicate Data Across Alibaba Cloud Accounts
This tutorial demonstrates data synchronization within a single Alibaba Cloud account. Select No.
RDS Instance ID
Select the ID of the source ApsaraDB RDS for MySQL instance.
Account
Enter the database account of the source ApsaraDB RDS for MySQL instance. The account must have the
REPLICATION CLIENT,REPLICATION SLAVE,SHOW VIEW, andSELECTpermissions.Database Password
Enter the password for the database account.
Encryption
Select Non-encrypted or SSL-encrypted based on your requirements. If you select SSL-encrypted, you must first enable SSL encryption on the ApsaraDB RDS for MySQL instance. For more information, see Enable SSL encryption.
Destination Database
Select Instance
Select an existing AnalyticDB for PostgreSQL in Serverless mode instance. This parameter is optional.
Database Type
Select AnalyticDB for PostgreSQL.
Access Method
Select Alibaba Cloud Instance.
Instance Region
Select the region where the destination AnalyticDB for PostgreSQL in Serverless mode instance is located.
Cluster ID
Select the ID of the destination AnalyticDB for PostgreSQL in Serverless mode instance.
Database Name
Enter the name of the database in the destination AnalyticDB for PostgreSQL in Serverless mode instance to which the objects will be synchronized.
Database Account
Enter the initial account of the destination AnalyticDB for PostgreSQL in Serverless mode instance.
NoteYou can also use an account that has the
RDS_SUPERUSERpermission. For more information about how to create an account, see Manage users and permissions.Database Password
Enter the password for the database account.
After the configuration is complete, click Test Connectivity and Proceed to configure synchronization objects and advanced settings.
NoteDTS automatically adds the CIDR blocks of DTS servers in the corresponding region to the whitelist of the ApsaraDB RDS for MySQL instance. You do not need to manually add them. For a list of CIDR blocks, see IP address blocks of DTS servers.
After the data synchronization task is complete or released, manually remove the CIDR blocks of the DTS servers from the whitelist.
Configure the task steps and synchronization objects.
Parameter
Description
Task Stages
Incremental Data Synchronization is selected by default. You must also select Schema Synchronization and Full Data Synchronization. After the precheck, DTS performs a full data synchronization of the selected objects from the source instance to the destination instance. This process creates a baseline for incremental data synchronization.
Processing Mode of Conflicting Tables
Precheck and Report Errors: DTS checks if the destination database contains tables with the same names as the source tables. If duplicates are found, the precheck fails and the task cannot start. Otherwise, the precheck passes.
NoteIf a table with a duplicate name in the destination database cannot be easily deleted or renamed, you can change the table name in the destination database. For more information, see Map object names.
Ignore Errors and Proceed: Skips the check for tables with the same names in the destination database.
WarningIf you select Ignore Errors and Proceed, data inconsistency may occur and pose risks to your business. Examples:
If the table schemas are identical and a record in the destination database has the same primary key value as a record in the source database:
During full data synchronization, DTS retains the record in the destination database. The record from the source database is not synchronized to the destination database.
During incremental data synchronization, DTS does not retain the record in the destination database. The record from the source database overwrites the record in the destination database.
If the table schemas are different, the synchronization may fail or only some columns may be synchronized.
DDL and DML Operations to Be Synchronized
Select the DDL or DML operations to synchronize at the instance level. For information about supported operations, see Supported SQL operations.
NoteTo select SQL operations at the database or table level, right-click a synchronization object in the Selected Objects box and select the desired SQL operations in the dialog box that appears.
Select Objects
In the Source Objects box, select an object and click
to move it to the Selected Objects box.NoteYou can select objects to be synchronized only at the table level.
Rename Databases and Tables
To change the name of a single synchronization object in the destination instance, right-click the object in the Selected Objects box. For more information, see Map a single database, table, or column name.
To change the names of multiple synchronization objects in the destination instance in batches, click Batch Edit in the upper-right corner of the Selected Objects box. For more information, see Map multiple database, table, or column names in batches.
Filter data to synchronize
You can set a WHERE clause to filter data. For more information, see Set filter conditions.
SQL operations to synchronize
Right-click an object in the Selected Objects box. In the dialog box that appears, select the DML and DDL operations to synchronize. For information about supported operations, see Supported SQL operations.
Click Next: Advanced Settings.
Parameter
Description
Set Alerts
Configure alerts to receive notifications if the task fails or if latency exceeds the specified threshold.
No: An alert is not configured.
Configure: An alert is configured. You must also set the alert threshold and specify alert contacts.
Replicate temporary tables generated during DMS online DDL operations
If you use Data Management Service (DMS) to perform online DDL changes on the source database, select whether to synchronize the data from the temporary tables generated during these changes.
Yes: Synchronizes data from the temporary tables that are generated by online DDL changes.
NoteIf a large amount of data is generated in the temporary tables, the data synchronization task may be delayed.
No: Does not synchronize data from the temporary tables. Only the original DDL data from the source database is synchronized.
NoteThis may cause tables to be locked in the destination database.
Retry Time for Failed Connections
If the data synchronization task fails to connect to a database, DTS continuously retries to connect. The default retry duration is 120 minutes. You can customize a retry duration from 10 to 1,440 minutes. We recommend that you set the duration to 30 minutes or more. If DTS reconnects to the source and destination databases within the specified retry duration, the data synchronization task automatically resumes. Otherwise, the task fails.
NoteIf multiple instances share a database, the shortest configured retry duration applies to all of them.
You are charged for the runtime of a data synchronization task during the connection retry period. We recommend that you customize the retry duration based on your business requirements, or release the DTS instance as soon as the source and destination database instances are released.
Enclose Object Names in Quotation Marks
Select whether to enclose destination object names in quotes. If you select Yes and one of the following conditions is met, DTS encloses the destination object names in single or double quotes during schema synchronization and incremental data synchronization:
The source database is case-sensitive and uses mixed case.
The source table name does not start with a letter and contains characters other than letters, digits, or specific special characters.
NoteThe supported special characters are underscores (
_), number signs (#), and dollar signs ($).The name of the schema, table, or column to be synchronized is a keyword, reserved word, or invalid character in the destination database.
NoteIf you select to enclose object names in quotes, you must use the quoted object names when you query data after the data synchronization is complete.
Configure ETL
Select whether to configure the ETL feature. For more information about ETL, see What is ETL?
Yes: Configures the ETL feature. You must enter data processing statements in the text box. For more information, see Configure ETL in a DTS migration or synchronization task.
No: Does not configure the ETL feature.
After completing the configuration, click Next: Configure Table And Field at the bottom of the page to set the primary key and distribution columns for the tables to be synchronized to the destination AnalyticDB for PostgreSQL.
Save the task and perform a precheck.
To view the parameters for configuring this instance via an API operation, hover over the Next: Save Task Settings and Precheck button and click Preview OpenAPI parameters in the tooltip.
If you have finished viewing the API parameters, click Next: Save Task Settings and Precheck at the bottom of the page.
NoteBefore a synchronization task starts, DTS performs a precheck. You can start the task only if the precheck passes.
If the precheck fails, click View Details next to the failed item, fix the issue as prompted, and then rerun the precheck.
If the precheck generates warnings:
For non-ignorable warning, click View Details next to the item, fix the issue as prompted, and run the precheck again.
For ignorable warnings, you can bypass them by clicking Confirm Alert Details, then Ignore, and then OK. Finally, click Precheck Again to skip the warning and run the precheck again. Ignoring precheck warnings may lead to data inconsistencies and other business risks. Proceed with caution.
When the Success Rate reaches 100%, click Next: Purchase Instance.
On the Purchase page, select the billing method and link specifications for the data synchronization instance. For more information, see the following table.
Category
Parameter
Description
New Instance Class
Billing Method
Subscription: You pay upfront for a specific duration. This is cost-effective for long-term, continuous tasks.
Pay-as-you-go: You are billed hourly for actual usage. This is ideal for short-term or test tasks, as you can release the instance at any time to save costs.
Resource Group Settings
The resource group to which the instance belongs. The default is default resource group. For more information, see What is Resource Management?.
Instance Class
DTS offers synchronization specifications at different performance levels that affect the synchronization rate. Select a specification based on your business requirements. For more information, see Data synchronization link specifications.
Subscription Duration
In subscription mode, select the duration and quantity of the instance. Monthly options range from 1 to 9 months. Yearly options include 1, 2, 3, or 5 years.
NoteThis option appears only when the billing method is Subscription.
Read and select the checkbox for Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start, and then click OK in the OK dialog box.
You can monitor the task progress on the data synchronization page.
FAQ
If a schema synchronization error persists after you confirm that the table schemas are consistent, Submit a ticket to contact technical support.
The
VACUUMoperation does not run automatically during data synchronization, which can reduce write performance. We recommend periodically runningVACUUMon your database.If an error occurs during full data synchronization, you must clear the data from the destination table and restart the synchronization process.
Instances in Serverless mode perform well with bulk writes to a single table but may perform poorly with hot data rows or multi-table, small-batch writes. For these scenarios, Submit a ticket for kernel parameter tuning.