Migrate data from RDS MySQL to Lindorm - Data Transmission Service

Prerequisites

Before you begin, make sure that you have:

A destination Lindorm instance that uses the wide table engine, with storage space larger than the storage space used by the source RDS MySQL instance. See Create an instance.
The MySQL-compatible connection endpoint enabled on the destination Lindorm instance. See Enable the MySQL-compatible feature.
A database (namespace) and a wide table created in the destination Lindorm instance to receive migrated data, with the wide table pre-partitioned based on the full data. Pre-partitioning determines how data is distributed across the Lindorm cluster. Choose a high-cardinality column as the partition key to distribute data evenly and avoid hot spots that would slow down reads and writes after migration. Poor partition key selection can also cause CDC-phase issues if updates affect partition key columns. For instructions on creating the database and table, see Connect to and use the wide table engine using the MySQL command line, Connect to and use the wide table engine using Lindorm-cli, Access the wide table engine using Lindorm Shell, and CREATE TABLE. For data type compatibility, see Data type mapping.

Objects you create must comply with the naming constraints and object limits for Lindorm instances.

Billing

Migration type	Task configuration fee	Data transfer fee
Full data migration	Free	Free, unless Access Method is set to Public IP Address, in which case standard Internet traffic fees apply. For details, see Billable items.
Incremental data migration	Charged. For details, see Billing overview.	—

Supported SQL operations for incremental migration

Operation type	SQL operations
DML	INSERT, UPDATE, DELETE
DDL	CREATE TABLE, DROP TABLE, ADD COLUMN

Due to LindormTable limitations, DTS does not migrate additional column attributes in ADD COLUMN statements. For example, if the source runs ALTER TABLE test ADD COLUMN col INT DEFAULT 0;, DTS runs ALTER TABLE test ADD COLUMN col INT; on the destination.

Database account permissions

Database	Full migration	Incremental migration
Source RDS MySQL	SELECT	Read and write
Destination Lindorm	Read and write on the destination namespace

If the source database account was not created and granted permissions in the RDS MySQL console, make sure the account has the REPLICATION CLIENT, REPLICATION SLAVE, SHOW VIEW, and SELECT permissions. To create an account and grant permissions via the console, see Create an account and Modify the permissions of an account.

For Lindorm account management, see User management.

Limitations

Source database limitations

The server that hosts the source database must have sufficient outbound bandwidth. Otherwise, the data migration speed is affected.
Tables to be migrated must have primary keys and unique fields. Without them, duplicate data may appear in the destination.
If you migrate at the table level and edit objects (such as mapping table or column names), a single task supports a maximum of 1,000 tables. If you exceed this limit, split the tables into multiple tasks or configure the task to migrate the entire database.
If the source is MySQL 8.0.23 or later and the data includes invisible columns, DTS cannot read those columns and data loss occurs. To make invisible columns visible before migrating, run:
```
ALTER TABLE <table_name> ALTER COLUMN <column_name> SET VISIBLE;
```
For more information, see Invisible Columns. Tables without explicit primary keys automatically generate invisible primary keys — make those visible too. See Generated Invisible Primary Keys.

Incremental migration only:

Binary logging must be enabled with the following settings. Without these, the precheck fails and the task cannot start. Binary log retention requirements: If binary logs are deleted before DTS can read them, the task may fail or data inconsistency may occur, and the DTS Service Level Agreement (SLA) may not apply.

ApsaraDB RDS for MySQL: at least 3 days (7 days recommended). See Configure parameters based on which the system automatically deletes the binary log files of an RDS instance.
Self-managed MySQL: at least 7 days.

Parameter	Required value	Effect if not set
`binlog_format`	`ROW`	Setting to `STATEMENT` or `MIXED` can cause data inconsistency in the destination.
`binlog_row_image`	`full`	Partial row images may cause incomplete data replication.
`log_slave_updates`	`ON`	Required only for self-managed MySQL in a dual-primary cluster. Without it, DTS cannot obtain all binary logs.

Full migration only:

Do not perform DDL operations on the source during full migration. DDL operations may cause the task to fail.

During full migration, DTS queries the source database and creates metadata locks, which can block DDL operations on the source.
Do not write new data to the source if you select only Full Data Migration. To keep source and destination in sync throughout migration, select both Full Data Migration and Incremental Data Migration.
Full data migration cannot be performed if the source RDS MySQL instance has the EncDB feature enabled.

RDS MySQL instances with Transparent Data Encryption (TDE) enabled support schema migration, full data migration, and incremental data migration.

Other limitations

Schema migration is not supported.
Migration of INDEX, PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, and foreign key objects is not supported.
Migration of BIT data type is not supported.
You cannot migrate only primary key fields. The data must contain at least one non-primary key field.
Data can only be written to the wide table engine of a Lindorm instance.
Empty strings of the VARBINARY type are treated as null by both DTS and Lindorm.
If the precision of a DECIMAL field in the destination differs from the source, the task fails.
Data written to Lindorm must comply with limits on data requests. Requests exceeding these limits cause the task to fail.
Full migration runs concurrent INSERT operations, which causes tablespace fragmentation in the destination. The destination tablespace will be larger than the source after full initialization completes.
Do not run online DDL tools such as pt-online-schema-change on migration objects in the source during migration. This causes the migration to fail.
If data is written to the destination by sources other than DTS during migration, data inconsistency may occur.
Data generated by binary log change operations — such as data restored from a physical backup or cascade operations — is not migrated to the destination.

If such data is not migrated, you can run a full migration again as long as your business is not affected.
If a DTS instance fails, the DTS helpdesk attempts to recover it within 8 hours. Recovery operations may include restarting the instance and adjusting DTS instance parameters (not database parameters). For parameters that may be adjusted, see Modify instance parameters.

Special cases

Self-managed MySQL source:

If a primary/secondary switchover occurs on the source while the migration task is running, the task fails.
DTS periodically runs CREATE DATABASE IF NOT EXISTS `test` on the source to advance the binary log file position.
If no DML operations are performed on the source for an extended period, migration latency readings may be inaccurate. Run a DML operation on the source to refresh the latency. If you select an entire database as the migration object, create a heartbeat table that receives updates every second.

ApsaraDB RDS for MySQL source:

Incremental migration only: A read-only RDS MySQL V5.6 instance cannot be used as the source because it does not record transaction logs.
DTS periodically runs CREATE DATABASE IF NOT EXISTS `test` on the source to advance the binary log file position.

Create a migration task

Step 1: Open the Data Migration page

Use one of the following methods:

DTS console

Log on to the DTS console.DTS console
In the left-side navigation pane, click Data Migration.
In the upper-left corner, select the region where the migration instance resides.

DMS console

Note

The exact navigation may vary depending on the mode and layout of the DMS console. For more information, see Simple mode and Customize the layout and style of the DMS console.

Log on to the DMS console.DMS console
In the top navigation bar, go to Data + AI > DTS (DTS) > Data Migration.
To the right of Data Migration Tasks, select the region from the drop-down list.

Step 2: Configure source and destination databases

Click Create Task.

Configure the source and destination databases using the following parameters:

Source database:

Parameter	Description
Task Name	The name of the DTS task. DTS generates a name automatically. Specify a descriptive name to make the task easy to identify. The name does not need to be unique.
Select Existing Connection	If the source instance is already registered with DTS, select it from the drop-down list — DTS populates the database parameters automatically. Otherwise, configure the parameters below manually.
Database Type	Select MySQL.
Access Method	Select Alibaba Cloud Instance.
Instance Region	Select the region where the source RDS MySQL instance resides.
Replicate Data Across Alibaba Cloud Accounts	Select No if the source is in the same Alibaba Cloud account.
RDS Instance ID	Select the ID of the source RDS MySQL instance.
Database Account	Enter the database account for the source instance. For required permissions, see Database account permissions.
Database Password	Enter the password for the database account.
Encryption	Select Non-encrypted or SSL-encrypted. To use SSL encryption, enable it on the RDS MySQL instance first. See Use a cloud certificate to enable SSL encryption.

Destination database:

Parameter	Description
Select Existing Connection	If the destination instance is already registered with DTS, select it from the drop-down list. Otherwise, configure the parameters below manually.
Database Type	Select Lindorm.
Access Method	Select Alibaba Cloud Instance.
Instance Region	Select the region where the destination Lindorm instance resides.
Instance ID	Select the ID of the destination Lindorm instance.
Database Account	Enter the database account for the destination instance.
Database Password	Enter the password for the database account.

Click Test Connectivity and Proceed.

Make sure that the CIDR blocks of DTS servers are added to the security settings of the source and destination databases. See Add DTS server IP addresses to a whitelist. If the source or destination is a self-managed database not accessed via Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Step 3: Configure objects to migrate

On the Configure Objects page, set the following parameters:

If you do not select Incremental Data Migration, do not write new data to the source during migration to ensure data consistency.

Parameter	Description
Migration Types	Select Full Data Migration for a one-time migration. Select both Full Data Migration and Incremental Data Migration to keep the destination in sync with the source throughout the migration.
Processing Mode of Conflicting Tables	Keep the default value.
Capitalization of Object Names in Destination Instance	Controls the capitalization of database names, table names, and column names in the destination. The default is DTS default policy. For more information, see Specify the capitalization of object names in the destination instance.
Source Objects	Select the databases or tables to migrate, then click to move them to Selected Objects.
Selected Objects	If database, table, or column names in the destination Lindorm instance differ from those in the source, use object name mapping. For details, see Map object names. To apply row-level filters, right-click a table and specify WHERE conditions. See Specify filter conditions. To select which SQL operations to include in incremental migration, right-click an object and select the operations.

Click Next: Advanced Settings and configure the following:

Parameter	Description
Dedicated Cluster for Task Scheduling	By default, DTS uses the shared cluster. For improved stability, purchase a dedicated cluster. See What is a DTS dedicated cluster.
Retry Time for Failed Connections	How long DTS retries if the source or destination is unavailable. Valid values: 10–1,440 minutes. Default: 720 minutes. Set this to more than 30 minutes. If DTS reconnects within this window, it resumes the task; otherwise, the task fails. Note: if multiple tasks share the same source or destination, the most recently set value applies. You are charged for the DTS instance during retries.
Retry Time for Other Issues	How long DTS retries failed DDL or DML operations. Valid values: 1–1,440 minutes. Default: 10 minutes. Set this to more than 10 minutes. This value must be smaller than Retry Time for Failed Connections.
Enable Throttling for Full Data Migration	Throttles full migration to reduce load on source and destination. Configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s). Available only when Full Data Migration is selected.
Enable Throttling for Incremental Data Migration	Throttles incremental migration. Configure RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s). Available only when Incremental Data Migration is selected.
Environment Tag	Select a tag to identify the instance. No tag is required for this example.
Whether to delete SQL operations on heartbeat tables of forward and reverse tasks	Controls whether DTS writes heartbeat table SQL operations to the source while the instance runs. Select Yesalert notification settings to skip writing (a latency value may appear on the DTS instance). Select No to write heartbeat operations (this may affect physical backup and cloning of the source).
Configure ETL	Select Yes to configure extract, transform, and load (ETL) rules in the code editor. See Configure ETL in a data migration or data synchronization task. Select No to skip ETL configuration.
Monitoring and Alerting	Select Yes to receive alerts when the task fails or migration latency exceeds the threshold. Configure the alert threshold and notification settings. See Configure monitoring and alerting when you create a DTS task.

Step 4: Run the precheck

Click Next: Save Task Settings and Precheck.

To view the OpenAPI parameters for this task configuration, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters before proceeding.

DTS runs a precheck before starting the migration. If the precheck fails:

Click View Details next to each failed item, resolve the issue, then click Precheck Again.
If a precheck alert can be ignored, click Confirm Alert Details, then Ignore in the dialog box, then OK, then Precheck Again. Ignoring alerts may result in data inconsistency.

Step 5: Purchase and start the instance

Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

On the Purchase Instance page, configure the following:

Parameter	Description
Resource Group	The resource group for the migration instance. Default: default resource group. See What is Resource Management?
Instance Class	DTS provides instance classes that vary in the migration speed. You can select an instance class based on your business scenario. See Instance classes of data migration instances.

Read and accept the Data Transmission Service (Pay-as-you-go) Service Terms, then click Buy and Start. Click OK in the confirmation message.

You can view the progress of the task on the Data Migration page.

If a data migration task cannot be used to migrate incremental data, the task automatically stops. The Completed status is displayed.

If a data migration task can be used to migrate incremental data, the task does not automatically stop. The incremental data migration task runs continuously. The Running status is displayed.