Migrate data from a PolarDB for MySQL cluster to Lindorm - Data Transmission Service

Prerequisites

Before you begin, ensure that you have:

A destination Lindorm instance that uses the wide table engine, with storage capacity greater than the space used by the source PolarDB for MySQL cluster. See Create an instance.
The MySQL compatibility feature enabled on the destination Lindorm instance. See Enable the MySQL compatibility feature.
A database (namespace) and a wide table created in the destination Lindorm instance. If needed, pre-partition the wide table based on expected data volume. See Connect to and use the wide table engine using a MySQL command-line tool, Connect to and use the wide table engine using Lindorm-cli, Access the wide table engine using Lindorm Shell, CREATE TABLE, and Data type mapping.

The objects you create must meet the requirements in Quotas and limits.

Limitations

The following limitations apply to this migration path. Each limitation is labeled with the migration type it affects: (Full), (Incremental), or (Both).

Source database

Limitation	Applies to
The source database server must have sufficient outbound bandwidth. Insufficient bandwidth reduces migration speed.	Both
Tables must have PRIMARY KEY or UNIQUE constraints, with all fields unique. Otherwise, the destination database may contain duplicate records.	Both
If you select tables as migration objects and need to rename tables or columns in the destination database, a single task supports up to 1,000 tables. For more than 1,000 tables, configure multiple tasks or migrate the entire database instead.	Both
Binary logging must be enabled and `loose_polar_log_bin` must be set to `on`. See Enable binary logging and Modify parameters. Note Enabling binary logging incurs storage charges for the space occupied by binary logs.	Incremental only
Binary logs must be retained for at least 3 days. A 7-day retention period is strongly recommended. Shorter retention may cause DTS to fail when fetching binary logs, resulting in task failure or data loss. Insufficient retention also means the DTS Service-Level Agreement (SLA) may not apply. See Modify the retention period.	Incremental only
Do not perform DDL operations on migration objects during migration. DTS queries the source database during full migration, creating metadata locks that can block DDL operations.	Full only
Do not write new data to the source database. To maintain real-time data consistency, run both full data migration and incremental data migration instead of full migration alone.	Full only (when run without incremental)

Other limitations

Limitation	Applies to
Schema migration is not supported.	Both
Migration of BIT data type columns is not supported.	Both
You cannot migrate only primary key fields. The migrated data must include at least one non-primary key field.	Both
DTS does not migrate read-only nodes of the source PolarDB for MySQL cluster.	Both
DTS does not migrate Object Storage Service (OSS) external tables from the source cluster.	Both
DTS writes data only to the wide table engine of the Lindorm instance.	Both
DTS does not support primary/standby switchover during full migration. If a switchover occurs, reconfigure the migration task promptly.	Full only
VARBINARY empty strings are treated as null values in both DTS and the Lindorm instance. For example, a source column value of `''` (empty VARBINARY) becomes `NULL` in the destination.	Both
If DECIMAL columns in the destination Lindorm instance have different precision than the source, the task fails.	Both
Data in the Lindorm instance must meet the requirements in Limits on data requests. Otherwise, the task fails.	Both
DTS consumes read and write resources on both the source and destination databases, increasing database load. Run migrations during off-peak hours when the CPU load of both databases is below 30%.	Full only
Concurrent INSERT operations during full migration cause table fragmentation in the destination database. After full migration, the storage used by destination tables is larger than in the source instance.	Full only
Do not use tools such as pt-online-schema-change to perform online DDL operations on migration objects in the source database during migration.	Both
If other sources write to the destination database during migration, data inconsistency may occur.	Both
If a DTS instance fails, DTS helpdesk attempts recovery within 8 hours. During recovery, operations such as restarting the instance or adjusting DTS instance parameters (not database parameters) may be performed. See Modify instance parameters for parameters that may be modified.	Both

Billing

Migration type	Link configuration fees	Data transfer fees
Full data migration	Free	Free when Access Method for the destination database is Alibaba Cloud Instance. If Access Method is Public IP Address, data transfer over the Internet incurs additional fees. See Billing overview.
Incremental data migration	Charged. See Billing overview.	—

SQL operations supported for incremental migration

Operation type	SQL statements
DML	INSERT, UPDATE, DELETE
DDL	CREATE TABLE, DROP TABLE, ADD COLUMN

Due to wide table engine limitations, extra attributes in ADD COLUMN statements are not migrated. For example, ALTER TABLE test ADD COLUMN col INT DEFAULT 0; on the source becomes ALTER TABLE test ADD COLUMN col INT; on the destination.

Required permissions

Database	Full migration	Incremental migration
Source PolarDB for MySQL cluster	SELECT	Read and write
Destination Lindorm instance	Read and write on the destination namespace

For step-by-step instructions on creating accounts and granting permissions:

PolarDB for MySQL: Create and manage database accounts
Lindorm: User management

Create a migration task

Go to the Data Migration page and select the region where the migration instance resides.
DTS console
1. Log on to the DTS console.
2. In the left-side navigation pane, click Data Migration.
3. In the upper-left corner of the page, select the region in which the data migration instance resides.
DMS console

Note
The DMS console layout may vary by mode. See Simple mode and Customize the layout and style of the DMS console.
1. Log on to the DMS console.
2. In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Data Migration.
3. From the drop-down list to the right of Data Migration Tasks, select the region in which the data synchronization instance resides.
Click Create Task.

Configure the source and destination databases.

Section	Parameter	Description
(None)	Task Name	A name for the DTS task. DTS generates a name automatically. Specify a descriptive name to identify the task easily. The name does not need to be unique.
Source Database	Select Existing Connection	If the instance is registered with DTS, select it from the drop-down list and DTS populates the remaining fields automatically. Otherwise, configure the fields below manually. In the DMS console, select the instance from Select a DMS database instance. See Manage database connections.
	Database Type	Select PolarDB for MySQL.
	Access Method	Select Alibaba Cloud Instance.
	Instance Region	Select the region of the source PolarDB for MySQL cluster.
	Replicate Data Across Alibaba Cloud Accounts	Select No if the source instance belongs to the current Alibaba Cloud account.
	PolarDB Cluster ID	Select the ID of the source PolarDB for MySQL cluster.
	Database Account	Enter the database account. See Required permissions.
	Database Password	Enter the password for the database account.
	Encryption	Select an encryption method as needed. For Secure Sockets Layer (SSL) encryption, see Configure SSL encryption.
Destination Database	Select Existing Connection	If the instance is registered with DTS, select it from the drop-down list. Otherwise, configure the fields below manually. See Manage database connections.
	Database Type	Select Lindorm.
	Access Method	Select Alibaba Cloud Instance.
	Instance Region	Select the region of the destination Lindorm instance.
	Instance ID	Select the ID of the destination Lindorm instance.
	Database Account	Enter the database account. See Required permissions.
	Database Password	Enter the password for the database account.

Click Test Connectivity and Proceed.

Make sure the CIDR blocks of DTS servers are added to the security settings (whitelists) of both the source and destination databases. See Add DTS server IP addresses to a whitelist. If the source or destination database uses an access method other than Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

On the Configure Objects page, select the objects to migrate and configure migration settings.

If you rename objects using object name mapping, other objects that depend on the renamed object may fail to migrate.

Parameter	Description
Migration Types	Select Full Data Migration for a one-time migration. Select both Full Data Migration and Incremental Data Migration to keep data consistent while your services remain online. If you select only full migration, do not write to the source database during migration.
Processing Mode of Conflicting Tables	Use the default value.
Capitalization of Object Names in Destination Instance	Controls how database, table, and column names are capitalized in the destination. The default is DTS default policy. See Specify the capitalization of object names in the destination instance.
Source Objects	Select objects to migrate and click to move them to Selected Objects. Databases, tables, and columns are all selectable.
Selected Objects	If the database (namespace), table, or column names in the destination differ from the source, use object name mapping. See Map object names. To filter rows by condition, right-click a table and specify a WHERE clause. See Specify filter conditions. To select specific SQL operations for incremental migration, right-click an object in Selected Objects.

Click Next: Advanced Settings and configure the following parameters.

Parameter	Description
Dedicated Cluster for Task Scheduling	By default, DTS schedules tasks to the shared cluster. Purchase a dedicated cluster for improved stability. See What is a DTS dedicated cluster.
Retry Time for Failed Connections	How long DTS retries when the source or destination database becomes unreachable. Valid values: 10–1,440 minutes. Default: 720 minutes. Set this to more than 30 minutes. If the databases reconnect within this window, DTS resumes the task; otherwise the task fails. If multiple tasks share the same source or destination, the most recently set value applies. DTS charges for the instance during retry.
Retry Time for Other Issues	How long DTS retries after DDL or DML failures. Valid values: 1–1,440 minutes. Default: 10 minutes. Set this to more than 10 minutes. This value must be smaller than Retry Time for Failed Connections.
Enable Throttling for Full Data Migration	Limits DTS read and write resource consumption during full migration to reduce database load. Configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s). Available only when Full Data Migration is selected.
Enable Throttling for Incremental Data Migration	Limits resource consumption during incremental migration. Configure RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s). Available only when Incremental Data Migration is selected.
Whether to delete SQL operations on heartbeat tables of forward and reverse tasks	Controls whether DTS writes heartbeat table operations to the source database. Yesalert notification settings: does not write heartbeat operations (a latency may be shown for the DTS instance). No: writes heartbeat operations (may affect features such as physical backup and cloning of the source database).
Environment Tag	(Optional) Select a tag to identify the instance.
Configure ETL	Whether to enable extract, transform, and load (ETL). Yes: opens a code editor for data processing statements. See Configure ETL in a data migration or data synchronization task. No: ETL is disabled. See What is ETL?
Monitoring and Alerting	Whether to configure alerts for task failures or latency thresholds. Yes: configure the alert threshold and notification contacts. See Configure monitoring and alerting when you create a DTS task. No: no alerts.

Click Next: Save Task Settings and Precheck. To preview the API parameters for this task, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters before proceeding.

DTS runs a precheck before starting the migration task. The task starts only after passing the precheck. - If a precheck item fails, click View Details next to it, address the issue, then click Precheck Again. - If an alert is triggered for a precheck item: - If the alert cannot be ignored, fix the issue and run the precheck again. - If the alert can be ignored, click Confirm Alert Details, then Ignore, then OK, then Precheck Again. Ignoring alerts may result in data inconsistency.
When Success Rate reaches 100%, click Next: Purchase Instance.

On the Purchase Instance page, configure the instance class.

Section	Parameter	Description
New Instance Class	Resource Group	The resource group for the migration instance. Default: default resource group. See What is Resource Management?
New Instance Class	Instance Class	The instance class determines migration speed. Select based on your requirements. See Instance classes of data migration instances.

Read the Data Transmission Service (Pay-as-you-go) Service Terms, select the check box to agree, then click Buy and Start. In the confirmation dialog box, click OK.

Verify the migration

Monitor task progress on the Data Migration page:

Full data migration only: The task stops automatically when complete. The status changes to Completed.
Full and incremental data migration: The incremental migration phase runs continuously and does not stop automatically. The status shows Running.