Data Transmission Service (DTS) supports full data migration and incremental data migration from an RDS MySQL instance to a PolarDB-X 1.0 instance. This topic uses an RDS MySQL instance as the source. The procedure is similar for other supported source database types.
Supported source databases
RDS MySQL instance
Self-managed MySQL databases:
Self-managed database with a public IP address
Self-managed database hosted on Elastic Compute Service (ECS)
Self-managed database connected over Express Connect, VPN Gateway, or Smart Access Gateway
Self-managed database connected over Database Gateway
Prerequisites
Before you begin, make sure that:
An RDS MySQL source instance is created. For more information, see Create an RDS MySQL instance.
A PolarDB-X 1.0 destination instance is created. For more information, see Create a PolarDB-X 1.0 instance.
The PolarDB-X 1.0 instance has more storage space than the RDS MySQL instance uses.
Required databases and tables are created in the PolarDB-X 1.0 instance. Schema migration is not supported — DTS cannot create the schema for you.
The database accounts for the source and destination databases have the required permissions. See Database account permissions.
Choose a migration type
DTS supports two migration types for this scenario. Select one or both based on your requirements.
| Migration type | What it does | Billing | Use when |
|---|---|---|---|
| Full data migration | Migrates all existing data from the source to the destination | Free for instance configuration. Internet traffic is charged when the Access Method of the destination database is set to Public IP Address. | You want a one-time migration and can tolerate brief write downtime |
| Incremental data migration | After full migration, continuously replicates new changes from the source | Charged | You need zero-downtime migration and want to keep the destination in sync until cutover |
If you select only full data migration, do not write new data to the source database during the migration. Otherwise, data inconsistency occurs between the source and destination databases. To maintain real-time data consistency, select both migration types.
Limitations
Review the following limitations before creating the migration task.
Source database requirements
The server of the source database must have sufficient outbound bandwidth. Insufficient bandwidth reduces migration speed.
Tables to be migrated must have primary keys or UNIQUE constraints with unique field values. Otherwise, duplicate data may exist in the destination database.
When selecting tables as migration objects with name mapping, a single task supports a maximum of 1,000 tables. If you exceed this limit, split the tables across multiple tasks or migrate the entire database instead.
The PolarDB-X 1.0 instance storage class must be RDS MySQL (private custom RDS). PolarDB for MySQL is not supported.
If the source database is MySQL 8.0.23 or later and the data includes invisible columns, those columns cannot be read and data loss occurs. To make the columns visible before migrating, run:
ALTER TABLE <table_name> ALTER COLUMN <column_name> SET VISIBLE;. Tables without primary keys automatically generate invisible primary keys — make those visible too. For more information, see Invisible Columns and Generated Invisible Primary Keys.
Incremental migration requirements
For incremental data migration, the source database must meet the following binary log requirements:
Binary logging is enabled.
binlog_formatis set toROW.binlog_row_imageis set toFULL.If the source is a self-managed MySQL database in a dual-primary cluster:
log_slave_updatesis set toON.Binary log retention period:
RDS MySQL instance: at least 3 days (7 days recommended). For more information, see Configure parameters based on which the system automatically deletes binary log files.
Self-managed MySQL database: at least 7 days.
If the binary log retention period is too short, DTS may fail to obtain binary logs, which can cause task failure or data loss.
Additional incremental migration limits:
Incremental migration does not support the utf8mb3 character set. Tasks involving this character set will fail.
A read-only RDS MySQL V5.6 instance cannot be used as the source for incremental migration because it does not record transaction logs.
If online DDL change operations using the temporary table mode are performed on the source database, data may be lost in the destination database.
Other limits
Do not perform DDL operations on the source database during full data migration. This can cause the migration task to fail.
During full data migration, DTS queries the source database, which creates metadata locks that may block DDL operations on the source database.
Full data migration causes table fragmentation in the destination database due to concurrent INSERT operations. After full migration, the storage space in the destination database will be larger than in the source instance.
DTS reads FLOAT and DOUBLE column values using
ROUND(COLUMN,PRECISION). Without a specified precision, DTS migrates FLOAT values with 38 digits of precision and DOUBLE values with 308 digits. Verify that this meets your requirements.DTS retries a failed migration task for up to 7 days. Before switching your application to the destination instance, end or release the task, or revoke the write permissions from the DTS database account to prevent destination data from being overwritten.
The data generated by change operations of binary logs, such as data restored from a physical backup or data from a cascade operation, is not recorded or migrated to the destination database while the migration task is running.
If the source database is an RDS MySQL instance with the EncDB feature enabled, full data migration cannot be performed.
RDS MySQL instances with Transparent Data Encryption (TDE) enabled support schema migration, full data migration, and incremental data migration.
If an instance fails, DTS helpdesk will attempt recovery within 8 hours. During recovery, DTS instance parameters may be adjusted. For more information, see Modify instance parameters.
Special cases
Self-managed MySQL source:
If a primary/secondary switchover occurs on the source database while the migration task is running, the task fails.
DTS executes
CREATE DATABASE IF NOT EXISTS \test\`` on the source database periodically to advance the binary log position.DTS calculates migration latency based on the latest migrated data timestamp in the destination and the current timestamp in the source. If no DML operation is performed on the source for a long time, the latency display may be inaccurate. Perform a DML operation on the source to refresh the latency.
If you select an entire database as the migration object, you can create a heartbeat table. The heartbeat table is updated or receives data every second.
RDS MySQL source:
DTS executes
CREATE DATABASE IF NOT EXISTS \test\`` on the source database periodically to advance the binary log position.
Database account permissions
Grant the required permissions to the database accounts before creating the migration task.
| Database | Full migration | Incremental migration |
|---|---|---|
| RDS MySQL instance | SELECT | REPLICATION CLIENT, REPLICATION SLAVE, SHOW VIEW, SELECT |
| PolarDB-X 1.0 instance | Read and write | Read and write |
For instructions on creating accounts and granting permissions:
RDS MySQL: Create an account and Modify account permissions
PolarDB-X 1.0: Manage database accounts
SQL operations supported for incremental migration
| Operation type | SQL statements |
|---|---|
| DML | INSERT, UPDATE, DELETE |
Create a migration task
Step 1: Go to the Data Migration page
Use one of the following methods:
DTS console
Log on to the DTS console.DTS console
In the left-side navigation pane, click Data Migration.
In the upper-left corner, select the region where the migration instance resides.
DMS console
The actual steps may vary based on the mode and layout of the DMS console. For more information, see Simple mode and Customize the layout and style of the DMS console.
Log on to the DMS console.DMS console
In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Data Migration.
From the drop-down list to the right of Data Migration Tasks, select the region where the migration instance resides.
Step 2: Configure source and destination databases
Click Create Task.
Review the Limits section at the top of the page to confirm the migration task can be created and run.
Configure the following parameters:
Task settings
| Parameter | Description |
|---|---|
| Task Name | Enter a descriptive name to identify the task. The name does not need to be unique. |
Source Database
| Parameter | Value |
|---|---|
| Select Existing Connection | If the source instance is registered with DTS, select it from the list. DTS auto-fills the remaining fields. Otherwise, configure the fields below. In the DMS console, select the instance from Select a DMS database instance. |
| Database Type | Select MySQL. |
| Access Method | Select Cloud Instance. |
| Instance Region | Select the region where the source RDS MySQL instance resides. |
| Cross-account | Select Not Cross-account for migrations within the same account. |
| RDS Instance ID | Select the ID of the source RDS MySQL instance. |
| Database Account | Enter the database account. See Database account permissions for required permissions. |
| Database Password | Enter the password for the database account. |
| Connection Method | Select Non-encrypted or SSL-encrypted. To use SSL encryption, enable it on the RDS MySQL instance first. For more information, see Use a cloud certificate to enable SSL encryption. |
Destination Database
| Parameter | Value |
|---|---|
| Select Existing Connection | If the destination instance is registered with DTS, select it from the list. DTS auto-fills the remaining fields. Otherwise, configure the fields below. In the DMS console, select the instance from Select a DMS database instance. |
| Database Type | Select PolarDB-X 1.0. |
| Access Method | Select Cloud Instance. |
| Instance Region | Select the region where the destination PolarDB-X 1.0 instance resides. |
| Instance ID | Select the destination PolarDB-X 1.0 instance. |
| Database Account | Enter the database account. See Database account permissions for required permissions. |
| Database Password | Enter the password for the database account. |
Click Test Connectivity and Proceed.
Make sure the CIDR blocks of DTS servers are added to the security settings of the source and destination databases. For more information, see Add DTS server IP addresses to a whitelist.
Step 3: Configure migration objects
On the Configure Objects page, configure the following settings:
| Parameter | Description |
|---|---|
| Migration Types | Select Full Data Migration for a one-time migration. Select both Full Data Migration and Incremental Data Migration to keep the destination in sync during cutover. |
| Processing Mode for Existing Destination Tables | Precheck and Report Errors: Checks for tables with the same name in the source and destination. If identical names exist, the precheck fails and the task cannot start. Use object name mapping to rename conflicting tables before migrating. Ignore Errors and Proceed: Skips the precheck. During full migration, conflicting records in the destination are retained. During incremental migration, conflicting records in the destination are overwritten. Proceed with caution. |
| Destination Database Object Name Case Policy | Controls the capitalization of database names, table names, and column names in the destination. By default, DTS default policy is selected. For more information, see Specify the capitalization of object names in the destination instance. |
| Source Objects | Select objects to migrate and click the arrow icon to add them to Selected Objects. Migration objects are selected at the table level. |
| Selected Objects | To rename a single object, right-click it in Selected Objects. See Map the name of a single object. To rename multiple objects at once, click Batch Edit. See Map multiple object names at a time. To set a row filter, right-click the table and configure a WHERE condition. See Set a filter condition. |
If you use object name mapping, other objects that depend on the renamed object may fail to migrate.
Click Next: Advanced Settings.
Step 4: Configure advanced settings
| Parameter | Description |
|---|---|
| Dedicated Cluster for Task Scheduling | By default, DTS schedules the task to the shared cluster. For improved stability, purchase a dedicated cluster. For more information, see What is a DTS dedicated cluster. |
| Retry Time for Failed Connections | How long DTS retries after a connection failure. Valid values: 10–1,440 minutes. Default: 720. Set to a value greater than 30. If DTS reconnects within this period, the task resumes. Otherwise, the task fails. Note When multiple tasks share the same source or destination database, the value set most recently takes effect. DTS charges for the instance during retries. |
| Retry Time for Other Issues | How long DTS retries after DDL or DML failures. Valid values: 1–1,440 minutes. Default: 10. Set to a value greater than 10. This value must be less than Retry Time for Failed Connections. |
| Enable Throttling for Full Data Migration | Limits DTS read/write resource usage during full migration to reduce database load. Configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s). Available only when Full Data Migration is selected. |
| Enable Throttling for Incremental Data Migration | Limits DTS resource usage during incremental migration. Configure RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s). Available only when Incremental Data Migration is selected. |
| Environment Tag | (Optional) Select a tag to identify the instance. |
| Whether to delete SQL operations on heartbeat tables of forward and reverse tasks | Controls whether DTS writes heartbeat table SQL operations to the source database. Yesalert notification settings: Does not write heartbeat operations. A latency may be displayed for the DTS instance. No: Writes heartbeat operations. Features such as physical backup and cloning of the source database may be affected. |
| Configure ETL | Whether to enable the extract, transform, and load (ETL) feature. Select Yes to enter data processing statements. For more information, see Configure ETL in a data migration or data synchronization task. Select No to skip ETL. |
| Monitoring and Alerting | Whether to configure alerts for the migration task. Select Yes to set alert thresholds and notification contacts. For more information, see Configure monitoring and alerting. |
Step 5: Save settings and run precheck
To preview the API parameters for this task configuration, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.
Click Next: Save Task Settings and Precheck.
DTS runs a precheck before the migration task starts. The task can only start after passing the precheck.
Precheck results fall into two categories:
| Result | Meaning | Action |
|---|---|---|
| Failed | The check item is not met | Click View Details, fix the issue, then click Precheck Again |
| Alert | The check item is partially met; the task can continue but may have risks | If the alert cannot be ignored: click View Details, fix the issue, then click Precheck Again. If the alert can be ignored: click Confirm Alert Details > Ignore > OK, then click Precheck Again. Ignored alerts may result in data inconsistency. |
Step 6: Purchase the instance
Wait until Success Rate reaches 100%, then click Next: Purchase Instance.
On the Purchase Instance page, configure the instance class:
Parameter Description Resource Group The resource group for the migration instance. Default: default resource group. For more information, see What is Resource Management? Instance Class Select an instance class based on your required migration speed. For more information, see Instance classes of data migration instances. Read and accept the Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start, then click OK in the confirmation message.
The task appears on the Data Migration page. Monitor progress there.
Full-only migration tasks stop automatically when complete. The status shows Completed.
Tasks that include incremental data migration do not stop automatically. The incremental phase runs continuously until you manually stop it. The status shows Running.
Cut over to the destination instance
After the incremental migration task reaches a steady state, cut over your application to the PolarDB-X 1.0 instance:
Stop all writes to the source RDS MySQL instance.
Wait for the migration latency to reach 0 seconds and the data gap to reach 0 KB. These two conditions must both be met before proceeding.
End or release the migration task in the DTS console, or revoke the write permissions from the DTS database account on the destination instance to prevent data from being overwritten if the task resumes automatically.
Switch your application connection strings to the PolarDB-X 1.0 instance.