Data Transmission Service:Migrate data from a self-managed TiDB database to a PolarDB for MySQL cluster

Source database limits

• Egress bandwidth: The source database server must have sufficient egress bandwidth. Insufficient bandwidth reduces migration speed.

• Primary key or UNIQUE constraint: Tables must have a PRIMARY KEY or UNIQUE constraint with all fields unique. Without these constraints, the destination database may contain duplicate records.

• Table limit per task: When migrating individual tables with column-level edits (such as renaming columns), a single task supports up to 1,000 tables. For more than 1,000 tables, split the work across multiple tasks or migrate the entire database instead.

• DDL restrictions during migration: During schema migration and full data migration, do not perform DDL operations that change the schema of the database or tables. The migration task fails if DDL operations are detected.

• Prefix indexes: TiDB metadata does not store prefix index lengths. After migration, prefix index lengths are lost, which may prevent the instance from running. Manually fix prefix index lengths for any affected tables after migration.

• Incremental data requirements: Incremental data migration requires a Kafka cluster and TiDB Binlog or TiCDC components. See (Optional) Set up incremental data collection.

Other limits

• Incremental data partition: DTS reads incremental data only from partition ID 0 in the Kafka topic.

• Position initialization: After creating a task with incremental migration, perform changes or insert test data on the source database promptly to update position information. Failing to do so may cause the task to fail due to excessive latency.

• 4-byte characters: If the data includes rare characters or emoji (4-byte characters), the destination database and tables must use the UTF8mb4 character set. If you use DTS schema migration, also set the character_set_server instance parameter to UTF8mb4 in the destination database.

• Full migration resource usage: Full data migration consumes read and write resources on both the source and destination databases, increasing their load. Evaluate database performance before migrating and run full migration during off-peak hours (for example, when CPU load is below 30%).

• Tablespace size after full migration: Concurrent INSERT operations during full data migration cause table fragmentation. The tablespace in the destination database will be larger than in the source database after migration completes.

• Writes to the destination during migration: Writing to the destination database during migration may cause data inconsistency between the source and destination.

• FLOAT and DOUBLE precision: DTS uses ROUND(COLUMN,PRECISION) to read FLOAT and DOUBLE values. The default precision is 38 digits for FLOAT and 308 digits for DOUBLE. Verify that these precision settings meet your requirements.

• Auto-resume: DTS automatically attempts to resume tasks that failed within the last seven days. Before switching workloads to the destination instance, stop or release the DTS task, or run REVOKE to remove write permissions from the DTS account on the destination instance.

• DDL failures: If a DDL statement fails in the destination database, the DTS task continues to run. View failed DDL statements in task logs. See View task logs.

• Technical support intervention: If a DTS task fails, technical support attempts to restore it within 8 hours. During restoration, the task may be restarted and task parameters (not database parameters) may be modified. For the parameters that may be modified, see the Modify instance parameters section.

Step 1: Set up a Kafka cluster

Use one of the following options:

• Self-managed Kafka cluster: Deploy Apache Kafka by following the Apache Kafka documentation.

  Warning

  To handle the volume of binary log data from TiDB, increase the following parameters in your Kafka configuration. See Kafka configuration.

  Component	Parameter	Why
  Broker	message.max.bytes	Sets the maximum size of a message the broker accepts. TiDB binary log events can be large, so the default limit causes write failures.
  Broker	replica.fetch.max.bytes	Sets the maximum bytes a follower can fetch per request. Must be at least as large as message.max.bytes to replicate oversized messages.
  Consumer	fetch.message.max.bytes	Sets the maximum bytes a consumer fetches per partition. Must be at least as large as message.max.bytes so DTS can read full events.

• Message Queue for Apache Kafka: Use the Alibaba Cloud managed Kafka service. See Getting started.

  Deploy the Message Queue for Apache Kafka instance in the same virtual private cloud (VPC) as the source database server to reduce network latency and ensure stable connectivity.

Step 2: Create a Kafka topic

Create a topic in the Kafka cluster or Message Queue for Apache Kafka instance.

Step 3: Configure TiDB Binlog or TiCDC

Choose one method based on your environment:

<details> <summary>Option A: TiDB Binlog</summary>

To minimize the impact of network latency, deploy the Pump component, Drainer component, and Kafka cluster on servers that share the same internal network as the source TiDB database server.

1. Deploy the Pump and Drainer components. See TiDB Binlog cluster deployment.

2. Edit the Drainer configuration file to set Kafka as the output. See Kafka custom development.Kafka Custom Development

3. Verify that the TiDB database server can connect to the Kafka cluster.

4. Add the CIDR blocks of DTS servers to the TiDB database whitelist. See Add the CIDR blocks of DTS servers.

</details>

<details> <summary>Option B: TiCDC</summary>

1. Install the TiCDC component. Use TiUP to add or scale out TiCDC on your existing TiDB cluster. See Deploy TiCDC.

2. Replicate incremental data to Kafka using tiup cdc cli changefeed create \ as the first command. See Replicate data to Kafka.

3. Verify that the TiDB database server can connect to the Kafka cluster.

</details>

Step 1: Go to the Data Migration page

Use one of the following methods:

Step 2: Configure source and destination databases

1. Click Create Task.

2. On the task configuration page, configure the source and destination databases.

Source database parameters

Destination database parameters

Step 3: Test connectivity

In the lower part of the page, click Test Connectivity and Proceed. In the CIDR Blocks of DTS Servers dialog box, click Test Connectivity.

Make sure the CIDR blocks of DTS servers are added to the security settings of the source and destination databases. See Add the CIDR blocks of DTS servers.

Step 4: Configure migration objects

1. On the Configure Objects page, set the following parameters.

Migration types and conflict handling

Object selection

Parameter	Description
Source Objects	Select the tables or databases to migrate. Click to move selected objects to Selected Objects.
Selected Objects	Right-click an object to rename it. See Map the name of a single object. Click Batch Edit to rename multiple objects at once. See Map multiple object names at a time. Right-click a table to configure a WHERE filter condition. Note Renaming an object may prevent dependent objects from being migrated. Note

1. Click Next: Advanced Settings.

Advanced settings

1. Click Next Step: Data Verification to configure data verification. See Configure a data verification task.

Step 5: Run the precheck

1. To preview the API parameters for this task, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

2. Click Next: Save Task Settings and Precheck.

DTS runs a precheck before starting the migration. The task only starts after the precheck passes.

For failed precheck items, click View Details to diagnose the issue, resolve it, then run the precheck again.

For alert items: if the alert cannot be ignored, fix the issue and rerun the precheck. If the alert can be ignored, click Confirm Alert Details, then click Ignore > OK > Precheck Again.

Step 6: Purchase and start the instance

1. Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the Purchase Instance page, configure the instance class.

1. Read and accept Data Transmission Service (Pay-as-you-go) Service Terms by selecting the check box.

2. Click Buy and Start, then click OK.

The task appears on the Data Migration page. Monitor its progress there.