Data Transmission Service:Migrate data from a self-managed TiDB database to AnalyticDB for MySQL 3.0

Choose a method

Configure Kafka parameters

Before connecting to DTS, increase the following Kafka parameters. Without these changes, the Kafka cluster may reject large binary log payloads from TiDB.

Prepare a Kafka cluster

Use one of the following options:

• Self-managed Kafka cluster — Deploy your own cluster. See the Apache Kafka official website.

• ApsaraMQ for Kafka instance — Create a managed instance. See Getting started overview. Deploy the instance in the same virtual private cloud (VPC) as the TiDB database server.

After the Kafka cluster is ready, create a topic with exactly one partition. DTS reads only from partition ID 0. Additional partitions cause data loss.

Use TiDB Binlog

1. Deploy Pump and Drainer on servers in the same internal network as the TiDB database. See TiDB Binlog cluster deployment.

2. Edit the Drainer configuration file to point to the Kafka cluster. See Binlog consumer client user guide.

3. Verify that the TiDB server can reach the Kafka cluster.

4. Add the CIDR blocks of DTS servers to the TiDB database allowlist. See Add the CIDR blocks of DTS servers.

Use TiCDC

1. Install TiCDC using TiUP to add a new TiCDC node or scale out an existing node in the TiDB cluster. See Deploy and maintain TiCDC.

2. Create a changefeed to replicate incremental data to Kafka. Use tiup cdc cli changefeed create as the starting command. See Replicate data to Kafka.

3. Verify that the TiDB server can reach the Kafka cluster.

Source database

• The TiDB server must have sufficient outbound bandwidth. Low bandwidth reduces migration speed.

• Tables to migrate must have PRIMARY KEY or UNIQUE constraints with all fields unique. Tables without these constraints may produce duplicate records in the destination.

• If you rename tables or columns during migration, a single task supports up to 1,000 tables. For more than 1,000 tables, split the migration into multiple tasks or migrate the entire database in one task.

• If you want to migrate incremental data from the source TiDB database, you must deploy a Kafka cluster and install related components for the TiDB database to collect the incremental data.

• Prefix indexes cannot be migrated. If the source database contains prefix indexes, the migration task may fail.

Incremental migration

• DTS reads data only from partition ID 0 of the Kafka topic. Configure the topic with exactly one partition.

• After creating an incremental migration task, perform operations on the source database or insert test data promptly to update the task's offset information. Long delays before the first data event can cause the task to fail.

Destination and task behavior

• Specify a custom primary key in the destination database or configure Primary Key Column in the Configurations for Databases, Tables, and Columns step. Missing primary key configuration causes data migration to fail.

• If the AnalyticDB for MySQL V3.0 cluster is being backed up while the DTS task runs, the task fails.

• If disk usage on AnalyticDB for MySQL V3.0 nodes exceeds 80%, the migration task is delayed and errors are returned. Estimate required disk space before starting.

• Full data migration increases load on both source and destination databases. Run migrations during off-peak hours when CPU load is below 30%.

• Full data migration uses concurrent INSERT operations, which cause fragmentation in destination tables. After full migration completes, the destination tablespace size is larger than the source.

• If other sources write to the destination during migration, data inconsistency may occur.

• DTS uses the ROUND(COLUMN,PRECISION) function to retrieve FLOAT and DOUBLE values. The default precision is 38 digits for FLOAT and 308 digits for DOUBLE. Verify that these defaults meet your requirements before starting.

• During schema migration, DTS does not support migrating materialized views to the destination AnalyticDB for MySQL instance. To use materialized views, manually create them in the destination instance after migration is complete.

• DTS automatically attempts to resume failed tasks for up to seven days. Before switching workloads to the destination, stop or release any failed tasks, or run the REVOKE statement to remove write permissions from the DTS accounts. Otherwise, a resumed task may overwrite destination data.

• If DDL statements fail in the destination, the DTS task continues running. View failed DDL statements in task logs. See View task logs.

• If a DTS task fails, DTS technical support attempts to restore it within 8 hours. During restoration, the task may be restarted and task parameters (not database parameters) may be modified. For a list of parameters that may change, see Modify instance parameters.

Step 1: Go to the Data Migration page

Use either the DTS console or the DMS console.

DTS console

1. Log on to the DTS console.

2. In the left-side navigation pane, click Data Migration.

3. In the upper-left corner, select the region where the migration instance will reside.

DMS console

Steps may vary based on the DMS console mode and layout. See Simple mode and Customize the layout and style of the DMS console.

1. Log on to the DMS console.

2. In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Data Migration.

3. From the drop-down list to the right of Data Migration Tasks, select the region where the migration instance will reside.

Step 2: Configure source and destination databases

1. Click Create Task.

2. Enter a Task Name. DTS generates a name automatically; specify a descriptive name to make the task easy to identify later.

3. Configure the source database:

   Parameter	Description
   Select Existing Connection	Select a registered database instance to auto-fill the parameters below, or leave blank and enter connection details manually.
   Database Type	Select TiDB.
   Access Method	Select the access method based on where TiDB is deployed. This example uses Self-managed Database on ECS. For other access methods, prepare the required environment first. See Preparation overview.
   Instance Region	The region where the TiDB database resides.
   ECS Instance ID	The ID of the ECS instance hosting TiDB.
   Port Number	The TiDB service port. Default: 4000.
   Database Account	The TiDB database account.
   Database Password	The password for the database account.
   Migrate Incremental Data	Select Yes if you want incremental data migration, then enter the Kafka cluster details in the Kafka cluster configuration section.

4. Configure the destination database:

   Parameter	Description
   Select Existing Connection	Select a registered instance to auto-fill the parameters below, or leave blank and enter connection details manually.
   Database Type	Select AnalyticDB for MySQL 3.0.
   Access Method	Select Alibaba Cloud Instance.
   Instance Region	The region where the destination cluster resides.
   Instance ID	The ID of the destination AnalyticDB for MySQL V3.0 cluster.
   Database Account	The database account for the destination cluster. See Required permissions.
   Database Password	The password for the database account.

5. Click Test Connectivity and Proceed. In the CIDR Blocks of DTS Servers dialog box, click Test Connectivity.

   DTS CIDR blocks must be added to the security settings of both source and destination databases. See Add the CIDR blocks of DTS servers.

Step 3: Configure objects to migrate

On the Configure Objects page, set the following parameters:

Click Next: Advanced Settings.

Step 4: Configure advanced settings

Step 5: Configure data verification (optional)

Click Next Step: Data Verification to set up a data verification task. See Configure a data verification task.

Step 6: Configure database and table fields (optional)

Click Next: Configure Database and Table Fields to set the Type, Primary Key Column, Distribution Key, Partition Key, Partitioning Rules, and Partition Lifecycle for destination tables.

This step is available only if Schema Migration is selected. Set Definition Status to All to view and edit all tables. To form a composite primary key, specify multiple columns in Primary Key Column and designate one or more as the Distribution Key and Partition Key. See CREATE TABLE.

Step 7: Run the precheck

Click Next: Save Task Settings and Precheck.

To view the API parameters for this task configuration, hover over the button and click Preview OpenAPI parameters before proceeding.

DTS runs a precheck before starting the migration. If the precheck fails:

• Click View Details next to the failed item, resolve the issue, and click Precheck Again.

• For alert items that can be safely ignored, click Confirm Alert Details > Ignore > OK, then click Precheck Again. Ignoring alerts may result in data inconsistency.

Step 8: Purchase an instance and start the task

1. Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the Purchase Instance page, configure the following:

   Parameter	Description
   Resource Group	The resource group for the migration instance. Default: default resource group. See What is Resource Management?.
   Instance Class	The instance class determines migration speed. See Instance classes of data migration instances.

3. Read and accept the Data Transmission Service (Pay-as-you-go) Service Terms.

4. Click Buy and Start, then click OK in the confirmation dialog.

Monitor task progress on the Data Migration page.

• Tasks without incremental migration stop automatically when complete. Status shows Completed.

• Tasks with incremental migration run continuously. Status shows Running. Stop the task manually after cutover.