Migrate data from a self-managed TiDB to RDS for MySQL - Data Transmission Service

This topic describes how to use Data Transmission Service (DTS) to migrate data from a self-managed TiDB database to an ApsaraDB RDS for MySQL instance.

Choose your migration path:

Path	Steps	When to use
Full migration only	Prerequisites → Procedure	One-time migration with a brief maintenance window
Full + incremental migration	Prerequisites → Preparations → Procedure	Zero-downtime migration; keep the source database live during migration

Important

Tables without a primary key or UNIQUE constraint may result in duplicate records in the destination database. Verify that all tables to be migrated have PRIMARY KEY or UNIQUE constraints before you begin.

Prerequisites

Before you begin, ensure that you have:

An ApsaraDB RDS for MySQL instance with available storage space larger than the total data size in the source TiDB database. For details, see Create an ApsaraDB RDS for MySQL instance.
(Full + incremental only) Completed the Preparations section to set up a Kafka cluster and configure TiDB Binlog or TiCDC for incremental data capture.

Preparations (incremental migration only)

Skip this section if you only need full data migration.

DTS reads incremental data from TiDB through a Kafka-based pipeline. DTS reads exclusively from partition 0 of the Kafka topic, so the topic must have exactly one partition. Before creating the DTS task, set up this pipeline using one of the following methods.

Method 1: TiDB Binlog

Deploy the source database server, Pump, Drainer, and the Kafka cluster on the same internal network to minimize network latency.

Prepare a Kafka cluster using one of these options:
- Self-managed Kafka cluster: See the Apache Kafka documentation. Set message.max.bytes and replica.fetch.max.bytes on the broker, and fetch.message.max.bytes on the consumer to values large enough to handle the TiDB binary log volume. For configuration details, see CONFIGURATION.
- ApsaraMQ for Kafka instance: See Getting started overview. Deploy the instance in the same virtual private cloud (VPC) as the source database server.
Create a topic in the Kafka cluster or ApsaraMQ for Kafka instance.
Important
The topic must have exactly one partition. DTS reads incremental data only from the partition whose ID is 0.
Deploy Pump and Drainer. For details, see TiDB Binlog cluster deployment.
Edit the Drainer configuration file to point to your Kafka cluster. For details, see Binlog consumer client user guide.
Verify that the TiDB database server can connect to the Kafka cluster.
Add the CIDR blocks of DTS servers to the TiDB database whitelist. For details, see Add the CIDR blocks of DTS servers.

Method 2: TiCDC

Prepare a Kafka cluster using one of these options:
- Self-managed Kafka cluster: See the Apache Kafka documentation. Set message.max.bytes and replica.fetch.max.bytes on the broker, and fetch.message.max.bytes on the consumer to values large enough to handle the TiDB binary log volume. For configuration details, see CONFIGURATION.
- ApsaraMQ for Kafka instance: See Getting started overview. Deploy the instance in the same VPC as the source database server.
Create a topic in the Kafka cluster or ApsaraMQ for Kafka instance.
Important
The topic must have exactly one partition. DTS reads incremental data only from the partition whose ID is 0.
Install TiCDC. We recommend that you use TiUP to add a new TiCDC node or scale out the existing TiCDC node in the TiDB cluster. For details, see Deploy and maintain TiCDC.
Replicate the incremental data in the source TiDB database to Kafka. We recommend that you use tiup cdc cli changefeed create \ in the first command line. For details, see Replicate data to Kafka.
Verify that the TiDB database server can connect to the Kafka cluster.

Permissions required

Database	Required permissions
TiDB database	SELECT on objects to migrate; SHOW VIEW
ApsaraDB RDS for MySQL instance	Read and write permissions on the destination database

For help creating and configuring accounts, see Create an account and Modify account permissions.

Billing

Migration type	Instance configuration fee	Internet traffic fee
Schema migration + full data migration	Free	Charged when Access Method is set to Public IP Address. See Billing overview.
Incremental data migration	Charged. See Billing overview.	—

SQL operations supported for incremental migration

Operation type	SQL statements
DML	INSERT, UPDATE, DELETE
DDL	CREATE TABLE, DROP TABLE, ALTER TABLE, RENAME TABLE, TRUNCATE TABLE, CREATE VIEW, DROP VIEW, ALTER VIEW

Usage notes

Source database limits:

The source database server must have sufficient outbound bandwidth. Insufficient bandwidth reduces migration speed.
Tables must have PRIMARY KEY or UNIQUE constraints with all fields unique. Tables without these constraints may cause duplicate records in the destination.
When migrating with table or column renaming, a single task supports up to 1,000 tables. For more than 1,000 tables, run multiple tasks in batches or migrate at the database level.
Incremental data migration requires a Kafka cluster with TiDB Binlog or TiCDC configured (see Preparations).

Full data migration:

Run migrations during off-peak hours when the CPU load on both databases is below 30%. DTS uses read and write resources on both databases during full migration.
Concurrent INSERT operations during full migration cause table fragmentation in the destination. Expect the destination tablespace to be larger than the source after migration completes.
Avoid writing data from other sources to the destination during migration to prevent data inconsistency.

Incremental data migration:

After creating the task, perform operations on the source database or insert test data promptly. This updates the offset information and prevents task failure from excessive latency.
DTS reads incremental data only from partition 0 of the Kafka topic.

Data type and character set considerations:

Data containing rare characters or emojis (4-byte characters) requires the destination tables to use the UTF8mb4 character set. If you use the schema migration feature, set the character_set_server parameter in the destination database to UTF8mb4.
Column names in MySQL are not case-sensitive. Writing column names that differ only in capitalization to the same destination table may produce unexpected results.
DTS uses ROUND(COLUMN, PRECISION) to retrieve FLOAT and DOUBLE values. Default precision: FLOAT = 38 digits, DOUBLE = 308 digits. Verify these precision settings meet your requirements.

After migration:

When the task Status shows Completed, run ANALYZE TABLE <table_name> to verify data was written to disk. An HA switchover in the destination MySQL database can cause data to exist only in memory, leading to data loss.
DTS attempts to resume failed tasks for up to 7 days. Before switching workloads to the destination database, stop or release any failed tasks, or run REVOKE to remove DTS write permissions on the destination. Otherwise, a resumed task may overwrite destination data with source data.
DDL statements that fail in the destination do not stop the DTS task. View failed DDL statements in the task logs. For details, see View task logs.
If a DTS task fails, DTS support will attempt to restore it within 8 hours. The task may be restarted and task parameters (not database parameters) may be modified during restoration. For the parameters that may be modified, see Modify instance parameters.
If the source database name does not comply with ApsaraDB RDS for MySQL naming conventions, create the destination database first and use the object name mapping feature to rename it during task configuration. For details, see Manage databases and Map object names.

Configure and run the migration task

Step 1: Go to the Data Migration page

Use one of these consoles:

DTS console:

Log on to the DTS console.
In the left-side navigation pane, click Data Migration.
In the upper-left corner, select the region where the migration instance resides.

Data Management Service (DMS) console:

The actual steps may vary based on the DMS console mode and layout. For details, see Simple mode and Customize the layout and style of the DMS console.

Log on to the DMS console.
In the top navigation bar, go to Data + AI > DTS (DTS) > Data Migration.
From the drop-down list next to Data Migration Tasks, select the region where the migration instance resides.

Step 2: Create the task and configure source and destination databases

Click Create Task.
Configure the source and destination databases using the following parameters:

Task settings:

Parameter	Description
Task Name	A name for the DTS task. DTS generates a name automatically. Specify a descriptive name to identify the task easily. The name does not need to be unique.

Source database:

Parameter	Description
Select Existing Connection	If the TiDB instance is registered with DTS, select it from the list and DTS populates the connection parameters automatically. Otherwise, configure the parameters below. In the DMS console, use the Select a DMS database instance list.
Database Type	Select TiDB.
Access Method	Select the access method based on where TiDB is deployed. This example uses Self-managed Database on ECS. For other access methods, complete the required environment setup first. See Preparation overview.
Instance Region	The region where the TiDB database resides.
ECS Instance ID	The ID of the ECS instance hosting the TiDB database.
Port Number	The TiDB service port. Default: 4000.
Database Account	The TiDB account. See Permissions required for the required permissions.
Database Password	The password for the TiDB account.
Migrate Incremental Data	Select Yes to enable incremental data migration, then enter the Kafka cluster information in the Kafka cluster parameters section. Select No for full migration only.

Destination database:

Parameter	Description
Select Existing Connection	If the RDS instance is registered with DTS, select it from the list. Otherwise, configure the parameters below. In the DMS console, use the Select a DMS database instance list.
Database Type	Select MySQL.
Access Method	Select Alibaba Cloud Instance.
Instance Region	The region where the destination ApsaraDB RDS for MySQL instance resides.
Replicate Data Across Alibaba Cloud Accounts	Select No for same-account migration.
RDS Instance ID	The ID of the destination ApsaraDB RDS for MySQL instance.
Database Account	The destination database account. See Permissions required.
Database Password	The password for the destination database account.
Encryption	Select Non-encrypted or SSL-encrypted. To use SSL encryption, enable SSL encryption on the RDS instance before configuring the task. See Use a cloud certificate to enable SSL encryption.

Step 3: Test connectivity

At the bottom of the page, click Test Connectivity and Proceed, then click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

DTS server CIDR blocks must be added (automatically or manually) to the security settings of both the source and destination databases. For details, see Add the CIDR blocks of DTS servers.

Step 4: Configure objects to migrate

On the Configure Objects page, set the following parameters:

Parameter	Description
Migration Types	Select the migration types based on your path: <br>- Schema Migration + Full Data Migration: Full migration only.<br>- Schema Migration + Full Data Migration + Incremental Data Migration: Full + incremental migration for zero-downtime migration.<br><br> Note If you skip Schema Migration, create the destination database and tables before starting the task, and enable object name mapping in Selected Objects. If you skip Incremental Data Migration, stop writing to the source database during migration to maintain data consistency.
Processing Mode of Conflicting Tables	Precheck and Report Errors (recommended): Fails the precheck if tables with identical names exist in both source and destination. Use object name mapping to rename conflicting tables. See Map object names. <br><br>Ignore Errors and Proceed: Skips the precheck for identical table names. Use with caution — this can cause data inconsistency. During full migration, existing records in the destination are retained. During incremental migration, existing records are overwritten. If schemas differ, only specific columns are migrated or the task may fail.
Capitalization of Object Names in Destination Instance	Controls the capitalization of database, table, and column names in the destination. DTS default policy is selected by default. See Specify the capitalization of object names.
Source Objects	Select tables or databases from the Source Objects section, then click to add them to Selected Objects.
Selected Objects	- To rename a single object, right-click it in Selected Objects. See Map the name of a single object.<br>- To rename multiple objects at once, click Batch Edit. See Map multiple object names at a time.<br>- To filter rows, right-click a table and specify filter conditions.<br><br> Note Renaming an object may break other objects that depend on it.

Step 5: Configure advanced settings

Click Next: Advanced Settings and configure the following parameters:

Parameter	Description
Retry Time for Failed Connections	How long DTS retries failed connections after the task starts. Valid values: 10–1,440 minutes. Default: 720. Set to at least 30 minutes. DTS resumes the task if reconnection succeeds within this window; otherwise, the task fails.<br><br> Note If multiple tasks share the same source or destination database, the most recently set retry time applies to all of them. DTS charges for instance usage during retries.
Retry Time for Other Issues	How long DTS retries failed DDL or DML operations. Valid values: 1–1,440 minutes. Default: 10. Set to at least 10 minutes. This value must be smaller than Retry Time for Failed Connections.
Enable Throttling for Full Data Migration	Limits resource usage during full migration to reduce load on the databases. Configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s). Available only when Full Data Migration is selected.
Enable Throttling for Incremental Data Migration	Limits resource usage during incremental migration. Configure RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s). Available only when Incremental Data Migration is selected.
Environment Tag	An optional tag to identify the DTS instance.
Configure ETL	Enable the extract, transform, and load (ETL) feature to transform data during migration. Select Yes and enter processing statements in the code editor. See What is ETL? and Configure ETL.
Monitoring and Alerting	Set up alerts for task failure or migration latency exceeding a threshold. Select Yes and configure the alert threshold and notification settings. See Configure monitoring and alerting.

Step 6: Configure data verification (optional)

Click Next Step: Data Verification to set up a data verification task. For details, see Configure a data verification task.

Step 7: Save settings and run the precheck

To preview the API parameters for configuring this task programmatically, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.
To proceed, click Next: Save Task Settings and Precheck.

DTS runs a precheck before the migration starts. The task cannot start until the precheck passes.

If the precheck fails, click View Details next to the failed item, resolve the issue, then click Precheck Again.

For precheck alerts: if the alert cannot be ignored, fix the issue and rerun the precheck. If it can be ignored, click Confirm Alert Details, then Ignore, then OK, then Precheck Again. Ignoring alerts may cause data inconsistency.

Step 8: Purchase the instance and start migration

Wait for Success Rate to reach 100%, then click Next: Purchase Instance.

On the Purchase Instance page, configure the following:

Parameter	Description
Resource Group	The resource group for the migration instance. Default: default resource group. See What is Resource Management?
Instance Class	The migration speed depends on the instance class. See Instance classes of data migration instances to select a class for your workload.

Read and accept Data Transmission Service (Pay-as-you-go) Service Terms by selecting the check box.
Click Buy and Start, then click OK in the confirmation dialog.

Monitor the task on the Data Migration page:

Full migration only: The task stops automatically when complete. The Status shows Completed.
Full + incremental migration: The task runs continuously. The Status shows Running.

Kafka cluster parameters

When Migrate Incremental Data is set to Yes, configure the Kafka cluster in the source database section:

Parameter	Description
Kafka Cluster Type	The deployment location of the Kafka cluster. This example uses Self-managed Database on ECS. If you select Express Connect, VPN Gateway, or Smart Access Gateway, also select a VPC from Connected VPC and specify Domain Name or IP.
Kafka Data Source Component	Select based on the method used in Preparations: Use the default binlog format of the TiDB database. (TiDB Binlog) or Use the TiCDC Canal-JSON format. (TiCDC).
ECS Instance ID	The ID of the ECS instance hosting the Kafka cluster.
Port Number	The Kafka service port.
Kafka Cluster Account	The Kafka username. Leave blank if authentication is not enabled.
Kafka Cluster Password	The Kafka password. Leave blank if authentication is not enabled.
Kafka Version	The Kafka version. If the version is 1.0 or later, select 1.0.
Encryption	Select Non-encrypted or SCRAM-SHA-256 based on your security requirements.
Topic	The topic that receives incremental data. The topic must have only one partition.

What's next

After the migration task status shows Completed or the incremental migration latency reaches zero, switch your application to the destination database:

Stop writes to the source TiDB database.
Wait for the incremental migration latency to reach 0 and the task to process all remaining changes.
Update your application connection strings to point to the ApsaraDB RDS for MySQL instance.
Run ANALYZE TABLE <table_name> on key tables to verify data integrity.
Stop or release the DTS migration task.