Data Transmission Service:Migrate data from an RDS for MySQL instance to a DataHub project

Source database requirements

• The source server must have enough outbound bandwidth. Low bandwidth reduces migration speed.

• Tables must have a PRIMARY KEY or UNIQUE constraint, and all fields must be unique. Without this, the destination may contain duplicate records.

• If you select tables as objects to migrate and need to rename tables or columns in the destination, a single task supports up to 1,000 tables. For more than 1,000 tables, split them across multiple tasks or migrate the entire database instead.

For incremental data migration, enable binary logging in the ApsaraDB RDS console and set the following parameters. If either value is incorrect, the precheck fails and the task cannot start.

Binary log retention requirements — if DTS cannot read a required binary log, the task fails and data may be lost or inconsistent:

See View and delete binary log files of an ApsaraDB RDS for MySQL instance.

During schema migration: Do not run DDL operations that change database or table schemas. Doing so causes the task to fail.

Other limitations

• Only tables can be selected as objects to migrate.

• Do not use tools such as pt-online-schema-change for DDL operations during migration — this may cause the task to fail.

• If only DTS writes to the destination, Data Management (DMS) can perform online DDL on source tables. See Perform lock-free DDL operations.

  Warning

  If other tools also write to the destination, do not use DMS for online DDL. Data loss in the destination may occur.

• DTS uses ROUND(COLUMN, PRECISION) to read FLOAT and DOUBLE columns. Default precision: 38 digits for FLOAT, 308 digits for DOUBLE. Verify that these defaults meet your requirements before starting.

• DTS retries failed tasks for up to 7 days. Before switching your workload to the destination, stop or release the task — or run REVOKE to remove DTS write permissions. Otherwise, source data may overwrite the destination after a retry.

Special cases for self-managed MySQL sources

• A primary/secondary failover during migration causes the task to fail.

• If no DML operations occur on the source for an extended period, migration latency readings may be inaccurate. Run a DML operation on the source to refresh the latency value. If you select an entire database as the migration object, create a heartbeat table that receives data every second.

• DTS periodically runs CREATE DATABASE IF NOT EXISTS `test` on the source to advance the binary log position.

Step 1: Go to the Data Migration Tasks page

1. Log on to the Data Management (DMS) console.

2. In the top navigation bar, click DTS.

3. In the left-side navigation pane, choose DTS (DTS) > Data Migration.

Console layout may vary. See Simple mode and Configure the DMS console based on your business requirements. Alternatively, go directly to the Data Migration Tasks page of the new DTS console.

Step 2: Select a region

From the drop-down list next to Data Migration Tasks, select the region where your data migration instance resides.

In the new DTS console, select the region in the upper-left corner.

Step 3: Configure source and destination databases

Click Create Task. Read the limits displayed at the top of the configuration page before proceeding.

Source database parameters:

Destination database parameters:

Step 4: Test connectivity

Click Test Connectivity and Proceed.

DTS automatically adds its CIDR blocks to the IP address whitelist of Alibaba Cloud database instances or the security group rules of Elastic Compute Service (ECS) instances. For self-managed databases hosted in data centers or on third-party clouds, manually add the DTS CIDR blocks to the database IP address whitelist. See Add the CIDR blocks of DTS servers to the security settings of on-premises databases.

Step 5: Select migration objects

Configure the following settings:

Parameter	Description
Synchronization Type	Select Schema Migration and/or Incremental Data Migration. Full Data Migration is not available for DataHub. If you do not select incremental data migration, avoid writing to the source during migration to maintain data consistency.
Processing Mode of Conflicting Tables	Precheck and Report Errors: checks for tables with identical names in source and destination. The task cannot start if conflicts exist. Use object name mapping to resolve conflicts without deleting destination tables. Ignore Errors and Proceed: skips the conflict check. During incremental migration, conflicting records overwrite existing destination records. If schemas differ, the task may fail or migrate only partial columns.
Apply New Naming Rules of Additional Columns	DTS adds system columns to the destination DataHub topic. If these column names conflict with existing topic columns, the task fails. Select Yes or No based on your topic structure. Check for conflicts before setting this parameter. See Modify the naming rules for additional columns.
Source Objects	Select objects and click to move them to Selected Objects. Only databases can be selected as objects.
Selected Objects	Right-click an object to rename it, specify WHERE filter conditions, or select the DML/DDL operations to migrate. Click Batch Edit to rename multiple objects at once. Renaming an object may break dependent objects.

Step 6: Configure advanced settings

Click Next: Advanced Settings and configure:

Step 7: Run the precheck

Click Next: Save Task Settings and Precheck. To preview the OpenAPI parameters before proceeding, hover over the button and click Preview OpenAPI parameters.

DTS runs a precheck before starting the task. The task can start only after passing all checks.

• If a check fails, click View Details next to the failed item, fix the issue, then click Precheck Again.

• If a check returns an alert that can be ignored, click Confirm Alert Details > Ignore > OK, then click Precheck Again.

Ignoring alert items may result in data inconsistency.

Step 8: Purchase the instance

Wait until the success rate reaches 100%, then click Next: Purchase Instance. On the Purchase Instance page, configure:

Step 9: Start the task

Read and accept the Data Transmission Service (Pay-as-you-go) Service Terms, then click Buy and Start.

The task appears in the task list. Monitor migration progress there.