Data Transmission Service:Migrate data from an RDS for MySQL instance to an ApsaraMQ for RocketMQ instance

Source database requirements

Migration objects

• Tables must have a primary key or UNIQUE constraint, with all fields unique. Without this, the destination may contain duplicate records.

• If you rename tables or columns at the destination, a single task can migrate up to 1,000 tables. Exceeding this limit causes a request error. Split the work across multiple tasks, or migrate at the database level instead of the table level.

Binary log requirements for incremental migration

To replicate incremental changes, the source MySQL instance must have binary logging enabled and configured as follows:

For the binlog retention period:

• RDS for MySQL instances: Store binary logs for at least 3 days; 7 days is recommended. For instructions, see Configure parameters based on which the system automatically deletes the binary log files of an RDS instance.

• Self-managed MySQL instances: Store binary logs for at least 7 days.

Insufficient retention may cause DTS to fail to obtain binary logs, which can result in task failure or, in exceptional cases, data inconsistency or loss. Retaining logs for less than the minimum period means the Service Level Agreement (SLA) guarantees may not apply.

DDL and DML restrictions

• Do not run DDL operations to change database or table schemas during the full data migration phase. DTS queries the source database during full migration, which creates metadata locks that can block DDL operations.

• If you run full data migration only (without incremental), do not write new data to the source during migration. To maintain real-time consistency, select both full data migration and incremental data migration.

Other source limitations

• Data changes recorded in binary logs—such as data restored from a physical backup or data from a cascade operation—are not migrated. If such data is missing at the destination, you can re-run a full migration as long as your business is not affected.

• If the source is MySQL 8.0.23 or later and the data includes invisible columns, those columns cannot be migrated and data loss occurs.

  • To make a column visible: ALTER TABLE <table_name> ALTER COLUMN <column_name> SET VISIBLE;

  • For details, see Invisible Columns and Generated Invisible Primary Keys.

Destination instance requirements

• Serverless ApsaraMQ for RocketMQ instances are not supported as a destination.

• The maximum message body size is 4 MB.

• If the destination instance is in a different region from the source, DTS accesses the destination through its public endpoint, which incurs data transfer costs. Enable public access for the destination instance before starting the task.

  • To check the internet access status: open the Instance Details page in the ApsaraMQ for RocketMQ console and check the Basic Information tab.

  • For pricing, see Internet traffic pricing (4.x series).

• Upgrading or downgrading the destination instance class causes messages to be delivered under a new rule configuration.

General limitations

• Do not write data to the destination from other sources during migration. External writes may cause data inconsistency.

• Do not use tools such as pt-online-schema-change to run online DDL operations on migration objects in the source during migration.

• Full data migration uses concurrent INSERT operations, which causes table fragmentation at the destination. After migration, table storage at the destination will be larger than at the source.

• Run migration during off-peak hours, when the CPU load of both databases is below 30%. DTS consumes read and write resources from both the source and destination during full data migration, which can increase database load.

• If the always-encrypted (EncDB) feature is enabled on the source RDS for MySQL instance, full data migration is not supported. Transparent Data Encryption (TDE) does not have this restriction—TDE-enabled instances support both full and incremental migration.

• If an instance fails, DTS technical support attempts to recover it within 8 hours. During recovery, the DTS instance parameters may be adjusted, but the source and destination database parameters are not modified.

Special cases for self-managed MySQL sources

• If a primary/secondary switchover occurs on the source while the migration task is running, the task fails.

• DTS executes CREATE DATABASE IF NOT EXISTS \test\`` on the source on a schedule to advance the binary log file position.

• Migration latency is calculated based on the timestamp of the latest migrated data at the destination compared to the current timestamp in the source. If no DML operations are performed on the source for an extended period, the displayed latency may be inaccurate. Run a DML operation on the source to refresh the latency value. If you are migrating an entire database, consider creating a heartbeat table that receives data every second.

Special cases for RDS for MySQL sources

• A read-only RDS for MySQL V5.6 instance (which does not record transaction logs) cannot be used as a source for incremental data migration.

• DTS executes CREATE DATABASE IF NOT EXISTS \test\`` on the source on a schedule to advance the binary log file position.

Step 1: Open the Data Migration page

Use one of the following methods to open the Data Migration page and select the region where the migration instance will reside.

DTS console

1. Log on to the DTS console.

2. In the left-side navigation pane, click Data Migration.

3. In the upper-left corner of the page, select the target region.

DMS console

The actual navigation path may vary based on your DMS console mode and layout. For more information, see Simple mode and Customize the layout and style of the DMS console.

1. Log on to the DMS console.

2. In the top navigation bar, go to Data + AI > DTS (DTS) > Data Migration.

3. From the drop-down list to the right of Data Migration Tasks, select the target region.

Step 2: Configure the source and destination

Click Create Task to open the task configuration page. Configure the source and destination as described in the following table.

Step 3: Test connectivity

Click Test Connectivity and Proceed.

Make sure that the CIDR blocks of DTS servers are added—automatically or manually—to the security settings of the source and destination databases. For more information, see Add DTS server IP addresses to a whitelist. If the source or destination is a self-managed database and Access Method is not set to Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Step 4: Select migration objects

On the Configure Objects page, configure the migration settings.

Step 5: Configure advanced settings

Click Next: Advanced Settings.

Step 6: Run the precheck

Click Next: Save Task Settings and Precheck.

Before you can start the migration task, DTS runs a precheck. The task starts only after the precheck passes.

If the precheck fails, click View Details next to each failed item, resolve the issue, and click Precheck Again.

If an alert is triggered: if the alert can be ignored, click Confirm Alert Details, then Ignore in the dialog box, and click Precheck Again. If the alert cannot be ignored, resolve the underlying issue before proceeding. Ignoring alerts may cause data inconsistency.

To view the API parameters for this configuration before saving, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

Step 7: Purchase the migration instance

1. Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the Purchase Instance page, configure the following parameters.

1. Read and accept Data Transmission Service (Pay-as-you-go) Service Terms by selecting the check box.

2. Click Buy and Start, then click OK in the confirmation dialog box.

After the task starts, track its progress on the Data Migration page:

• Full data migration only: the task stops automatically when complete. The status shows Completed.

• Full and incremental data migration: the incremental migration phase runs continuously and does not stop automatically. The status shows Running.