Data Transmission Service:Migrate data from RDS MySQL to a self-managed Doris database

Required permissions

To create and grant permissions:

• RDS for MySQL: see Create an account and Modify the permissions of an account.

• Doris: see Authentication and authorization.

Binary log requirements (incremental migration only)

For incremental migration, the source RDS for MySQL instance must have binary logging configured correctly:

• binlog_format = ROW

• binlog_row_image = FULL

• Local binary log retention: at least 3 days (7 days recommended)

If the source is a self-managed MySQL database, local binary logs must be retained for at least 7 days.

If either setting is wrong, the precheck fails and migration cannot start.

For instructions on setting the binary log retention period for RDS for MySQL, see Delete local logs automatically.

Source database limitations

• The server hosting the source database must have sufficient outbound bandwidth. Low bandwidth reduces migration speed.

• If you perform incremental migration on a self-managed dual-primary MySQL cluster (where each node is the primary for the other), enable the log_slave_updates parameter on the source.

• To migrate tables with column-level edits (such as column name mapping), a single task supports a maximum of 1,000 tables. For larger migrations, split tables across multiple tasks or migrate the entire database.

• Do not perform DDL operations that change the database or table schema during schema migration or full data migration. Doing so fails the task.

• If you run full data migration only (without incremental), do not write new data to the source during migration.

• Data generated by operations not recorded in binary logs — such as physical backup restoration or cascade operations — is not migrated.

• If the source is MySQL 8.0.23 or later and the data contains invisible columns, those columns are not migrated, which may cause data loss. Run ALTER TABLE <table_name> ALTER COLUMN <column_name> SET VISIBLE; to make invisible columns visible before migration. Tables without a primary key automatically generate an invisible primary key — you must also make this invisible primary key visible. For more information, see Generated Invisible Primary Keys.

• RDS for MySQL instances with the always-confidential feature enabled do not support full data migration. Instances with Transparent Data Encryption (TDE) enabled support all migration types.

Destination (Doris) limitations

• DTS can only write to tables using the Unique Key model or Duplicate Key model in Doris.

• When a destination table uses the Duplicate Key model, DTS converts UPDATE and DELETE statements to INSERT statements. Duplicate rows may appear if the migration task is retried, restarted, or if multiple DML operations target the same record between syncs. Use the additional columns _is_deleted, _version, and _record_id to deduplicate. See Additional column information for details.

• The only configurable table parameter in DTS is bucket_count (number of buckets). The value must be a positive integer; the default is auto.

• Do not create a new cluster in the destination Doris database during migration. If the task fails for this reason, restart the migration instance.

• Do not add Backend (BE) nodes to the Doris cluster during migration. If the task fails for this reason, restart the migration instance.

• Doris only accepts database and table names that start with a letter. Use object name mapping to rename any objects that do not start with a letter.

• If any database, table, or column name contains Chinese characters, use object name mapping to rename it (for example, to an English name). Otherwise, the task may fail.

• Avoid DDL operations that modify multiple columns simultaneously or modify the same table in quick succession.

• In multi-table merge scenarios (multiple source tables merged into one destination table), the source tables must have identical schemas.

• MySQL VARCHAR(M) uses character length (M); Doris VARCHAR(N) uses byte length (N). If you skip schema migration and create Doris tables manually, set the Doris VARCHAR length to four times the MySQL VARCHAR length.

• When using Data Management (DMS) or gh-ost for online DDL on the source, DTS migrates only the original DDL statements, not the temporary table data. This may cause table locking on the destination. pt-online-schema-change is not supported for online DDL on the source — using it may result in data loss or task failure.

• Do not use tools such as pt-online-schema-change on migration objects in the source database during migration.

• Full data migration uses concurrent INSERT operations, which causes table fragmentation. After full migration, the destination tables may use more storage than the source.

• If another source writes to the destination database during migration, data inconsistency may occur.

Incremental migration behavior

By default, DTS batches incremental writes to the destination at most once every 5 seconds per synchronization object. This results in a normal synchronization latency of up to 10 seconds.

To reduce latency, adjust the selectdb.reservoir.timeout.milliseconds parameter for the DTS instance in the console. Valid range: 1,000–10,000 milliseconds.

Shorter batch intervals increase write frequency to the destination, which increases load and response time (RT) on Doris and may raise the overall synchronization latency. Adjust based on destination load capacity.

Performance recommendations

Before starting migration, verify that the CPU load of both the source and destination databases is below 30%. Run the migration during off-peak hours to minimize impact on production workloads.

Special cases

If the source is a self-managed MySQL database:

• A primary/secondary failover on the source fails the migration task.

• DTS calculates latency by comparing the UNIX timestamp of the last migrated record against the current time. If no DML operations run on the source for an extended period, the displayed latency may be inaccurate. Run a DML operation on the source to refresh it, or create a heartbeat table that is written to every second.

• DTS periodically runs CREATE DATABASE IF NOT EXISTS `test` on the source to advance the binary log offset.

• For Amazon Aurora MySQL or other clustered MySQL instances, confirm that the domain name or IP address configured in the task always resolves to the read/write (RW) node.

If the source is an RDS for MySQL instance:

• RDS for MySQL read-only instances (such as read-only instances of RDS for MySQL 5.6) do not record transaction logs and cannot be used as the source for incremental migration.

• DTS periodically runs CREATE DATABASE IF NOT EXISTS `test` on the source to advance the binary log offset.

Step 1: Open the Data Migration page

Use one of the following methods to open the Data Migration page.

DTS console

1. Log on to the DTS console.

2. In the left-side navigation pane, click Data Migration.

3. In the upper-left corner, select the region where the migration instance resides.

Data Management (DMS) console

The exact navigation path may vary based on the DMS console mode and layout. For details, see Simple mode and Customize the layout and style of the DMS console.

1. Log on to the DMS console.

2. In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Data Migration.

3. From the drop-down list next to Data Migration Tasks, select the region.

Step 2: Configure source and destination databases

Click Create Task to open the task configuration page, then configure the parameters below.

Step 3: Test connectivity

At the bottom of the page, click Test Connectivity and Proceed, then click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Make sure the DTS server CIDR blocks can be added (automatically or manually) to the security settings of both the source and destination. For details, see Add DTS server IP addresses to a whitelist.

Step 4: Select migration objects and types

On the Configure Objects page, configure the following settings.

Migration types

Processing mode of conflicting tables

Capitalization of object names in destination instance

By default, DTS default policy is applied. Adjust if you need object name capitalization to match the source or destination. See Specify the capitalization of object names in the destination instance.

Selecting and configuring migration objects

1. In Source Objects, select databases or tables to migrate, then click to move them to Selected Objects.

2. In Selected Objects, right-click any object to:

   • Map its name to a different name in the destination

   • Select specific incremental SQL operations at the database or table level

   • Set WHERE conditions to filter rows (Set filter conditions)

   • Set the bucket_count parameter (if schema migration is selected): right-click a table, go to Parameter Settings, set Enable Parameter Settings to Yes, enter the value, and click OK

Using object name mapping may cause dependent objects to fail migration.

Step 5: Configure advanced settings

Click Next: Advanced Settings and configure the parameters below.

Step 6: (Optional) Configure database and table fields

Click Next: Configure Database and Table Fields to set the Primary Key Column, Distribution Key, and Engine for destination tables.

This step is available only if Schema Migration is selected.

Set Definition Status to All to view and modify all tables.

Primary Key Column supports composite primary keys. Select one or more columns from Primary Key Column as the Distribution Key.

For tables without a primary key or UNIQUE constraint, set Engine to duplicate.

Step 7: Run the precheck

Click Next: Save Task Settings and Precheck.

To preview OpenAPI parameters before saving, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

DTS runs a precheck before starting migration. The task only starts after the precheck passes.

• If any item fails, click View Details next to the failed item, resolve the issue, and click Precheck Again.

• If a non-critical alert appears and can be safely ignored, click Confirm Alert Details > Ignore > OK > Precheck Again. Ignoring alerts may result in data inconsistency.

Step 8: Purchase and start the instance

1. Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the Purchase Instance page, configure the instance:

   Parameter	Description
   Resource Group	Resource group for the migration instance (default: default resource group). See What is Resource Management?
   Instance Class	Determines migration speed. Choose based on your requirements. See Instance classes of data migration instances.

3. Read and accept the Data Transmission Service (Pay-as-you-go) Service Terms.

4. Click Buy and Start, then click OK in the confirmation dialog.

Verify migration status

View task progress on the Data Migration page.

• Full migration only: the task stops automatically when complete. Status shows Completed.

• Full + incremental migration: the incremental phase runs continuously. Status shows Running.

If an instance fails, DTS helpdesk attempts recovery within 8 hours. Recovery operations may include restarting the instance and adjusting DTS instance parameters (not database parameters). For details, see Modify instance parameters.