Data Transmission Service (DTS) can synchronize an entire MySQL instance — including all databases and tables — to an ApsaraDB RDS for MySQL instance in a single task. The task runs schema synchronization, full data synchronization, and incremental data synchronization automatically, with no object selection required.
This feature is in canary release and is available only to specific customers.
Limitations
Review these limitations before configuring a task.
Scope restrictions
Source database type: MySQL only.
Destination database type: MySQL only.
Destination access method: Alibaba Cloud Instance only.
Two-way synchronization DTS instances are not supported. You cannot use a two-way synchronization DTS instance to synchronize an entire instance. A one-way synchronization instance configured for entire-instance sync cannot be converted to a two-way instance.
Synchronization type is fixed: schema synchronization, full data synchronization, and incremental data synchronization are all enabled and cannot be changed.
The following configuration options are unavailable: selecting specific objects, mapping object names, filtering by SQL statement type or WHERE clause, adding columns, and configuring the extract, transform, and load (ETL) feature.
Data verification is not supported.
Objects to be synchronized cannot be modified after the task starts.
The destination database version must be the same as or later than the source. Earlier versions may cause compatibility issues.
Performance and data consistency
During initial full data synchronization, DTS uses read and write resources of both databases. Run the task during off-peak hours, when CPU usage on both databases is below 30%.
Concurrent INSERT operations during initial full data synchronization cause table fragmentation in the destination. After full synchronization completes, the destination tablespace may be larger than the source.
Write to the destination only through DTS while synchronization is running. Writes from other sources may cause data inconsistency.
If DDL operations fail in the destination, the task continues running. Check task logs for details. For more information, see View task logs.
After the task Status changes to Completed, run
ANALYZE TABLE <table_name>on the destination to verify data integrity. In some cases (for example, after a high-availability (HA) switchover in the source), data may only be written to memory, resulting in data loss.
Encryption
If the source is an ApsaraDB RDS for MySQL instance with EncDB enabled, full data synchronization is not supported.
If the source is an ApsaraDB RDS for MySQL instance with Transparent Data Encryption (TDE) enabled, schema synchronization, full data synchronization, and incremental data synchronization are all supported.
Task recovery
If a DTS task fails, DTS support will attempt to restore it within 8 hours. The task may be restarted and its parameters may be modified during restoration. Database parameters are not modified. The parameters that may be modified include but are not limited to the parameters described in the Modify the parameters of a DTS instance topic.
Foreign key behavior
During schema synchronization, DTS synchronizes foreign keys from the source to the destination.
During full and incremental data synchronization, DTS temporarily disables foreign key constraint checks and cascade operations at the session level. Cascade update or delete operations on the source during this period may cause data inconsistency.
Special cases for self-managed MySQL
A primary/secondary switchover on the source while the task is running causes the task to fail.
DTS calculates synchronization latency based on the timestamp of the latest synchronized data in the destination and the current timestamp in the source. If no DML operations occur on the source for an extended period, the displayed latency may be inaccurate. Run a DML operation on the source to refresh the latency reading.
Consider creating a heartbeat table that is updated or receives data every second. This keeps the latency reading accurate when the source database is otherwise idle.
DTS periodically runs
CREATE DATABASE IF NOT EXISTS 'test'on the source to advance the binary log file position.
Special cases for ApsaraDB RDS for MySQL
Read-only RDS MySQL 5.6 instances that do not record transaction logs cannot be used as the source.
DTS periodically runs
CREATE DATABASE IF NOT EXISTS 'test'on the source to advance the binary log file position.
Prerequisites
Before you begin, make sure that:
If the source MySQL database is deployed in a data center or on another cloud platform, it is connected to Alibaba Cloud. For more information, see Preparation overview.
The destination ApsaraDB RDS for MySQL instance is created and has enough free storage space to hold all data from the source. For more information, see Create an ApsaraDB RDS for MySQL instance.
The source server has sufficient outbound bandwidth. Insufficient bandwidth limits synchronization speed.
The source database meets the binary log requirements described in the table below.
For supported source and destination database versions, see Overview of data synchronization scenarios.
Binary log requirements
| Parameter | Required value | Notes |
|---|---|---|
| Binary logging | Enabled | Enabled by default on ApsaraDB RDS for MySQL. Must be enabled manually on self-managed MySQL databases. |
binlog_format | ROW | Required for self-managed MySQL. Not applicable to RDS MySQL (managed automatically). |
binlog_row_image | FULL | Required for both RDS MySQL and self-managed MySQL. If not set to FULL, the precheck fails and the task cannot start. For RDS MySQL, see Modify instance parameters. |
log_slave_updates | ON | Required only for self-managed MySQL in a dual-primary cluster. Ensures DTS can obtain all binary logs. For more information, see Create an account for a self-managed MySQL database and configure binary logging. |
| Binary log retention | At least 7 days | If logs are purged before DTS reads them, the task may fail or data inconsistency may occur. For RDS MySQL, see Manage binary log files. |
During schema synchronization and full data synchronization, do not run DDL statements that change database or table schemas. Doing so causes the task to fail.
Billing
| Synchronization type | Fee |
|---|---|
| Schema synchronization and full data synchronization | Free |
| Incremental data synchronization | Charged. For more information, see Billing overview. |
SQL operations that can be synchronized
| Operation type | SQL statements |
|---|---|
| DML | INSERT, UPDATE, DELETE |
| DDL | ALTER TABLE, ALTER VIEW, CREATE DATABASE, CREATE FUNCTION, CREATE INDEX, CREATE PROCEDURE, CREATE TABLE, CREATE VIEW, DROP DATABASE, DROP INDEX, DROP TABLE, RENAME TABLE, TRUNCATE TABLE |
Required database account permissions
| Role | Database type | Required permissions |
|---|---|---|
| Source | Self-managed MySQL | SELECT on objects to be synchronized; REPLICATION CLIENT; REPLICATION SLAVE; SHOW VIEW; CREATE on databases and tables. For setup instructions, see Create an account for a self-managed MySQL database and configure binary logging. |
| Source | ApsaraDB RDS for MySQL | Read and write permissions. For setup instructions, see Create an account and Modify account permissions. |
| Destination | ApsaraDB RDS for MySQL | Read and write permissions. For setup instructions, see Create an account and Modify account permissions. |
If the source is an ApsaraDB RDS for MySQL instance and the account was not created through the RDS console, make sure the account has REPLICATION CLIENT, REPLICATION SLAVE, SHOW VIEW, and SELECT permissions.
To synchronize source database account information, the source and destination accounts need additional permissions. For more information, see Migrate database accounts.
Configure a synchronization task
Step 1: Open the Data Synchronization page
Use either console to start.
DTS console
Log on to the DTS console.
In the left-side navigation pane, click Data Synchronization.
In the upper-left corner, select the region where your synchronization instance resides.
DMS console
The steps may vary depending on the DMS console mode and layout. For more information, see Simple mode and Customize the layout and style of the DMS console.
Log on to the DMS console.
In the top navigation bar, move the pointer over Data + AI and choose DTS (DTS) > Data Synchronization.
On the right side of Data Synchronization Tasks, select your region from the drop-down list.
Step 2: Create a task
Click Create Task to open the task configuration page.
If the New Configuration Page button appears in the upper-right corner, click it to switch to the new version. If Back to Previous Version appears instead, you are already on the new version.
Some parameters differ between the new and previous versions of the configuration page. Use the new version.
Step 3: Configure source and destination databases
Fill in the parameters for both databases.
Task name
| Parameter | Description |
|---|---|
| Task Name | DTS generates a name automatically. Specify a descriptive name to make the task easy to identify. Task names do not need to be unique. |
Source database
| Parameter | Description |
|---|---|
| Select Existing Connection | Select an existing database connection to auto-populate the parameters below, or leave it blank to enter the details manually. To register a new database, see Manage database connections (DTS console) or Register an Alibaba Cloud database instance and Register a database hosted on a third-party cloud service or a self-managed database (DMS console). |
| Database Type | Select MySQL. |
| Access Method | Select the access method that matches your network setup. In this example, Express Connect, VPN Gateway, or Smart Access Gateway is selected. |
| Instance Region | The region where the virtual private cloud (VPC) of the source database resides. |
| Replicate Data Across Alibaba Cloud Accounts | Select No if the source uses the same Alibaba Cloud account as the destination. |
| Connected VPC | The ID of the VPC connected to the source MySQL instance. |
| Domain Name or IP | The IP address or domain name of the source database server. |
| Port Number | The service port of the source database. Default: 3306. |
| Database Account | The account for the source database. For required permissions, see Required database account permissions. |
| Database Password | The password for the database account. |
| Encryption | Select Non-encrypted or SSL-encrypted. To use SSL encryption, enable SSL on the RDS instance before configuring this task. For more information, see Use a cloud certificate to enable SSL encryption. |
Destination database
| Parameter | Description |
|---|---|
| Select Existing Connection | Select an existing database connection to auto-populate the parameters below, or leave it blank to enter the details manually. To register a new database, see Manage database connections (DTS console) or Register an Alibaba Cloud database instance and Register a database hosted on a third-party cloud service or a self-managed database (DMS console). |
| Database Type | Select MySQL. |
| Access Method | Select Alibaba Cloud Instance. |
| Instance Region | The region where the destination RDS MySQL instance resides. |
| Replicate Data Across Alibaba Cloud Accounts | Select No if the destination uses the same Alibaba Cloud account as the source. |
| RDS Instance ID | The ID of the destination ApsaraDB RDS for MySQL instance. |
| Database Account | The account for the destination database. For required permissions, see Required database account permissions. |
| Database Password | The password for the database account. |
| Encryption | Select Non-encrypted or SSL-encrypted. To use SSL encryption, enable SSL on the RDS instance before configuring this task. For more information, see Use a cloud certificate to enable SSL encryption. |
Step 4: Test connectivity
Click Test Connectivity and Proceed at the bottom of the page.
DTS must be able to access the source and destination databases. Make sure the CIDR blocks of DTS servers are added to the security settings of both databases. For more information, see Add the CIDR blocks of DTS servers.
If the source database access method is not Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.
Step 5: Configure synchronization objects
On the Configure Objects page, set Select Objects to Synchronize Entire Instance, then configure the remaining parameters.
| Parameter | Description |
|---|---|
| Synchronization Types | Schema Synchronization, Full Data Synchronization, and Incremental Data Synchronization are selected by default and cannot be changed. |
| Method to Migrate Triggers in Source Database | Select how triggers are synchronized. If no triggers exist, skip this parameter. For more information, see Synchronize or migrate triggers from the source database. This parameter is available only when Schema Synchronization is selected. |
| Enable Migration Assessment | Select Yes to check whether schemas of the source and destination databases (such as index length, stored procedures, and dependent tables) meet requirements. The precheck duration increases, but the assessment results do not affect the precheck outcome. This parameter appears only when Schema Synchronization is selected. |
| Synchronization Topology | One-way Synchronization is selected by default and cannot be changed. |
| Processing Mode of Conflicting Tables | Precheck and Report Errors (default): checks for tables with identical names in the source and destination. If duplicates exist, the precheck fails. To resolve this, use object name mapping to rename destination tables. For more information, see Map object names. Ignore Errors and Proceed: skips the duplicate-name check. During full data synchronization, existing records with matching primary or unique keys are retained in the destination. During incremental synchronization, they are overwritten. If schemas differ, initialization may fail or only partial columns may be synchronized. Use with caution. |
| Capitalization of Object Names in Destination Instance | Controls the capitalization of database names, table names, and column names in the destination. Default: DTS default policy. For more information, see Specify the capitalization of object names in the destination instance. |
| Selected Objects | Retain the default settings. |
Step 6: Configure advanced settings
Click Next: Advanced Settings and configure the following parameters.
| Parameter | Description |
|---|---|
| Dedicated Cluster for Task Scheduling | By default, DTS schedules the task to the shared cluster. To improve stability, purchase a dedicated cluster. For more information, see What is a DTS dedicated cluster. |
| Copy the temporary table of the Online DDL tool that is generated in the source table to the destination database. | Applies if you use DMS or gh-ost for online DDL operations on the source. Yes: synchronize data from temporary tables generated by online DDL operations (may extend task duration). No, Adapt to DMS Online DDL: synchronize only the original DDL operations run via DMS, not the temporary table data (destination tables may be locked). No, Adapt to gh-ost: synchronize only the original DDL operations run via gh-ost (destination tables may be locked; supports filtering shadow tables using default or custom regular expressions). Do not use pt-online-schema-change for online DDL operations on the source — this causes the task to fail. |
| Whether to Migrate Accounts | Select Yes to synchronize source database account information. If selected, choose the accounts to synchronize and verify their permissions. |
| Retry Time for Failed Connections | How long DTS retries failed connections before marking the task as failed. Valid values: 10–1440 minutes. Default: 720 minutes. Set this to at least 30 minutes. If multiple tasks share the same source or destination database, the shortest retry setting takes precedence. DTS charges for the instance during retries. |
| Retry Time for Other Issues | How long DTS retries failed DDL or DML operations before marking the task as failed. Valid values: 1–1440 minutes. Default: 10 minutes. Set this to at least 10 minutes. Must be less than the Retry Time for Failed Connections value. |
| Enable Throttling for Full Data Migration | Limit the load on the destination during full data synchronization by configuring Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s). Appears only when Full Data Synchronization is selected. |
| Enable Throttling for Incremental Data Synchronization | Limit the load on the destination during incremental synchronization by configuring RPS of Incremental Data Synchronization and Data synchronization speed for incremental synchronization (MB/s). |
| Whether to delete SQL operations on heartbeat tables of forward and reverse tasks | Controls whether DTS writes heartbeat SQL operations to the source database. Yes: DTS does not write heartbeat operations (a synchronization latency may be displayed). No: DTS writes heartbeat operations (physical backup and cloning of the source may be affected). |
| Environment Tag | An optional tag to identify the environment (for example, production or staging). |
| Monitoring and Alerting | No: no alerts. Yes: sends alert notifications when the task fails or synchronization latency exceeds the threshold. Configure the alert threshold and notification settings. For more information, see Configure monitoring and alerting. |
Step 7: Skip data verification
Click Next Step: Data Verification.
Data verification is not supported for entire-instance synchronization. This step is skipped automatically.
Step 8: Save settings and run the precheck
Click Next: Save Task Settings and Precheck.
To preview the API parameters for this task configuration, hover over the button and click Preview OpenAPI parameters before proceeding.
DTS runs a precheck before the task starts. The task only starts after the precheck passes.
If the precheck fails, click View Details next to each failed item, resolve the issue, and click Precheck Again.
If a precheck alert is triggered: if the alert can be ignored, click Confirm Alert Details, then click Ignore in the View Details dialog box, click OK, and click Precheck Again. Ignored alerts may result in data inconsistency.
Step 9: Purchase a synchronization instance
Wait until Success Rate reaches 100%, then click Next: Purchase Instance.
On the buy page, configure the instance parameters.
| Parameter | Description |
|---|---|
| Billing Method | Subscription: pay upfront for a fixed term; more cost-effective for long-term use. Pay-as-you-go: billed hourly; suitable for short-term use. Release the instance when no longer needed to avoid ongoing charges. |
| Resource Group Settings | The resource group for the synchronization instance. Default: default resource group. For more information, see What is Resource Management?. |
| Instance Class | DTS offers multiple instance classes with different synchronization speeds. For more information, see Instance classes of data synchronization instances. |
| Subscription Duration | Available only for the Subscription billing method. Options: 1–9 months, or 1, 2, 3, or 5 years. |
Read and select Data Transmission Service (Pay-as-you-go) Service Terms, then click Buy and Start. In the dialog box that appears, click OK.
The task appears in the task list. Track its progress there.