Data Transmission Service:Migrate data from an ApsaraDB RDS for MySQL instance to an ApsaraMQ for Kafka instance

Source database

Binary log requirements for incremental migration

Before enabling incremental data migration, verify and configure these MySQL settings:

For self-managed MySQL instances, you can fix non-compliant settings by running the corresponding command:

-- Set binlog format
SET GLOBAL binlog_format = 'ROW';

-- Set binlog row image
SET GLOBAL binlog_row_image = 'FULL';

Other limitations

• You can select only tables as the objects to be migrated.

• Migrate during off-peak hours to reduce load on the source and destination servers.

• Before switching production workloads to the destination, stop or release any DTS tasks that failed in the last 7 days. DTS attempts to resume failed tasks automatically, which could overwrite data in the destination. Alternatively, revoke write permissions from the DTS accounts on the destination.

• If data from other sources is written to the destination during migration, data inconsistency will occur.

• If the destination Kafka cluster is upgraded or downgraded during migration, restart the migration task.

• If Transparent Data Encryption (TDE) is enabled on the source RDS MySQL instance, schema migration, full data migration, and incremental data migration are all supported. If the EncDB feature is enabled instead, full data migration is not supported.

• If a DTS task fails, DTS technical support will attempt to restore it within 8 hours. During restoration, the task may be restarted and task-level parameters may be modified. Database parameters are not changed.

Special cases

Self-managed MySQL:

• If a primary/secondary switchover occurs while the task is running, the task fails.

• Migration latency may appear inaccurate if no DML operations run on the source for a long time. Run any DML operation on the source to refresh the latency. If you migrated an entire database, create a heartbeat table that receives updates every second.

• DTS periodically runs CREATE DATABASE IF NOT EXISTS \test\`` on the source to advance the binary log position.

ApsaraDB RDS for MySQL:

• Read-only RDS MySQL V5.6 instances (which do not record transaction logs) cannot be used as the source for incremental data migration.

• DTS periodically runs CREATE DATABASE IF NOT EXISTS \test\`` on the source to advance the binary log position.

DTS Avro

Records follow the DTS Avro schema definition. See the schema on GitHub.

Canal JSON

Records are written in Canal JSON format. The following example shows a typical INSERT event:

{
  "data": [
    {
      "id": "1",
      "name": "example_value",
      "updated_at": "2026-01-01 00:00:00"
    }
  ],
  "database": "example_db",
  "es": 1735689600000,
  "id": 1,
  "isDdl": false,
  "mysqlType": {
    "id": "int(11)",
    "name": "varchar(255)",
    "updated_at": "datetime"
  },
  "old": null,
  "pkNames": ["id"],
  "sql": "",
  "sqlType": {
    "id": 4,
    "name": 12,
    "updated_at": 93
  },
  "table": "example_table",
  "ts": 1735689600123,
  "type": "INSERT"
}

For UPDATE events, the old field contains the before-image values (the column values before the change).

For full parameter details and sample consumer code, see Canal JSON data format.

Step 1: Go to the Data Migration page

Use either console:

Step 2: Configure the source and destination databases

1. Click Create Task.

   Warning

   After configuring the source and destination, read the Limits displayed at the top of the page before proceeding. Skipping this step may cause the task to fail or data inconsistency.

2. Configure the parameters in the following table.

Source Database

Destination Database

1. Click Test Connectivity and Proceed.

   DTS must be able to access the source and destination. Add the CIDR blocks of DTS servers to the security settings of both databases. See Add the CIDR blocks of DTS servers. If the source or destination is a self-managed database with an access method other than Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Step 3: Select migration objects and configure settings

1. On the Configure Objects page, set the following parameters.

1. Click Next: Advanced Settings.

Step 4: Run the precheck

1. To preview the API parameters for this task, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

2. Click Next: Save Task Settings and Precheck. DTS runs an automated precheck before starting the task. If any item fails:

   • Click View Details next to the failed item, fix the issue, then click Precheck Again.

   • If an alert item is non-critical and can be ignored, click Confirm Alert Details > Ignore > OK, then click Precheck Again. Ignoring alerts may lead to data inconsistency.

Step 5: Purchase the instance and start the task

1. Wait for Success Rate to reach 100%, then click Next: Purchase Instance.

2. On the Purchase Instance page, configure the following:

1. Read and accept Data Transmission Service (Pay-as-you-go) Service Terms by selecting the check box.

2. Click Buy and Start, then click OK in the confirmation dialog.