Data Transmission Service:Migrate data from a PolarDB-X 1.0 instance to a Message Queue for Apache Kafka instance

Source database requirements

Hard constraints — migration fails if not met:

• Tables to be migrated must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination may contain duplicate records.

• Migration from a read-only PolarDB-X 1.0 instance is not supported.

• If you select tables as migration objects and need to rename tables or columns, a single task supports up to 1,000 tables. For more than 1,000 tables, split the migration into multiple tasks or migrate the entire database instead.

Required configuration for incremental data migration:

• Binary logging must be enabled, and binlog_row_image must be set to full. If this is not configured, the precheck fails and the task cannot start.

• Binary log retention periods: If the retention period is insufficient, DTS may fail to read the binary logs, which can cause task failure or data inconsistency. After full migration completes, you can reduce the retention period to more than 24 hours.

  • Incremental migration only: retain binary logs for more than 24 hours.

  • Full + incremental migration: retain binary logs for at least 7 days.

Operational restrictions during migration:

• Do not upgrade or downgrade the source instance, modify frequently-updated tables, change shard keys, or run DDL operations on source objects. These actions will cause the task to fail.

• During full and incremental migration, DTS temporarily disables foreign key constraint checks and cascade operations at the session level. Cascade updates or deletes during this period may cause data inconsistency.

• If the network type of the PolarDB-X 1.0 instance changes during migration, update the network connection settings in the DTS migration task accordingly.

• For full-only migration (without incremental): do not write data to the source during migration, as this will cause inconsistency. To keep data consistent, select Schema Migration, Full Data Migration, and Incremental Data Migration together.

Other limitations

• Evaluate the impact on source and destination database performance before migrating. Run migration tasks during off-peak hours. Full data migration uses read and write resources on both databases, which may increase server load.

• Full data migration uses concurrent INSERT operations, which causes table fragmentation in the destination. After full migration, the destination tablespace is larger than the source.

• DTS attempts to resume failed tasks for up to 7 days. Before switching workloads to the destination, stop or release any failed tasks. Alternatively, run the REVOKE statement to revoke write permissions from the DTS accounts that access the destination. If a failed task resumes after you switch workloads, source data may overwrite destination data.

Precautions

• DTS periodically updates the ` dts_health_check.ha_health_check ` table in the source database to advance the binary log position.

• If the destination ApsaraMQ for Kafka instance is upgraded or downgraded during migration, restart the instance to resume migration.

Step 1: Go to the Data Migration Tasks page

1. Log on to the Data Management (DMS) console.

2. In the top navigation bar, click DTS.

3. In the left-side navigation pane, choose DTS (DTS) > Data Migration.

Step 2: Select a region

From the drop-down list next to Data Migration Tasks, select the region where the migration instance resides.

Step 3: Configure source and destination databases

Click Create Task and configure the following parameters.

Source database (PolarDB-X 1.0)

Destination database (ApsaraMQ for Kafka)

Step 4: Test connectivity

Click Test Connectivity and Proceed at the bottom of the page.

For Alibaba Cloud database instances (such as ApsaraDB RDS for MySQL or ApsaraDB for MongoDB), DTS automatically adds its server CIDR blocks to the instance IP address whitelist. For self-managed databases on Elastic Compute Service (ECS) instances, DTS automatically adds the CIDR blocks to the ECS security group rules. If the self-managed database spans multiple ECS instances, manually add the CIDR blocks to each instance's security group rules. For self-managed databases in a data center or hosted by a third-party provider, manually add the DTS server CIDR blocks to the database IP address whitelist. For details, see Add the CIDR blocks of DTS servers to the security settings of on-premises databases.

Step 5: Select objects and configure migration settings

Configure the following parameters:

Step 6: Configure advanced settings

Click Next: Advanced Settings and configure the following:

Step 7: Run the precheck

Click Next: Save Task Settings and Precheck.

DTS runs a precheck before the task starts. The task can proceed only after the precheck passes.

• If the precheck fails, click View Details next to each failed item, fix the issues, and run the precheck again.

• If an alert appears during the precheck:

  • For alerts that cannot be ignored: click View Details, fix the issues, and rerun the precheck.

  • For alerts that can be ignored: click Confirm Alert Details, then click Ignore > OK > Precheck Again. Ignoring alerts may result in data inconsistency.

Step 8: Purchase the migration instance

Wait until the success rate reaches 100%, then click Next: Purchase Instance.

On the Purchase Instance page, configure the following:

Step 9: Start migration

Read and accept the Data Transmission Service (Pay-as-you-go) Service Terms, then click Buy and Start.

The migration task starts. Monitor progress in the task list.