Migrate data from PolarDB-X 2.0 to MaxCompute - Data Transmission Service

Use Data Transmission Service (DTS) to migrate data from a PolarDB-X 2.0 instance to a MaxCompute project. DTS supports one-time full migration and continuous incremental migration, so you can choose a strategy based on your acceptable downtime.

How migration works

DTS supports three migration types, which you can combine:

Migration type	What it transfers	Supports incremental CDC
Schema migration	Table definitions and indexes	No
Full data migration	All existing rows at a point in time	No
Incremental data migration	Change data (INSERT, UPDATE, DELETE, ADD COLUMN) after full migration completes	Yes

Naming conventions in MaxCompute

DTS uses specific naming conventions for tables created in MaxCompute:

Schema migration: DTS adds the _base suffix to the source table name. For example, if the source table is named customer, the destination table in MaxCompute is named customer_base. The destination table suffixed with _base is known as a full baseline table.
Full data migration: Historical data is migrated to the corresponding _base table (for example, customer → customer_base). This serves as the baseline for subsequent incremental synchronization.
Incremental migration: DTS creates an incremental log table in MaxCompute. The table name follows the format DestinationTableName_log (for example, customer_log). Incremental data is migrated to this table in real time.

Choose a migration strategy

Goal	Migration types to select
One-time transfer with acceptable downtime	Schema migration + Full data migration
Continuous sync with minimal downtime	Schema migration + Full data migration + Incremental data migration

Important

If you select full data migration only, stop all writes to the source database during migration to prevent data inconsistency between the source and destination. To ensure data consistency, we recommend that you select schema migration, full data migration, and incremental data migration as the migration types.

SQL operations supported by incremental data migration

Operation type	Supported statements
DML	INSERT, UPDATE, DELETE
DDL	ADD COLUMN (ADD COLUMN operations that include attribute columns are not supported)

Prerequisites

Before you begin, make sure that you have:

Activated MaxCompute and created a MaxCompute project. See Activate MaxCompute and Create a MaxCompute project.
Configured the MaxCompute whitelist to allow DTS to access MaxCompute. See Configure a whitelist for Alibaba Cloud services to access MaxCompute.
Created an AccessKey pair for the Alibaba Cloud account that owns the destination MaxCompute project. See Create an AccessKey pair.

Note

Alternatively, you can create a RAM user and set that RAM user as the super administrator for the MaxCompute project.

Limitations

Review these limitations before starting the migration task.

Source database limitations

Tables must have PRIMARY KEY or UNIQUE constraints, with all fields unique. Tables without these constraints may produce duplicate records in MaxCompute.
Read-only instances of Enterprise Edition PolarDB-X 2.0 are not supported.
If you rename tables or columns during migration (object name mapping), a single task supports up to 1,000 tables. For more than 1,000 tables, split into multiple tasks or migrate the entire database instead.
TABLEGROUP and databases or schemas with the Locality attribute are not supported.
Tables with reserved word names (for example, select) are not supported.

Binlog requirements for incremental data migration

If you include incremental data migration, configure the following binlog settings on the source PolarDB-X 2.0 instance before starting:

Setting	Required value	Notes
Binary logging	Enabled	Required for DTS to read change data
`binlog_row_image`	`full`	DTS rejects any other value during precheck
Binary log retention (incremental-only tasks)	> 24 hours	Insufficient retention may cause task failure or data loss; DTS SLA does not apply
Binary log retention (full + incremental tasks)	>= 7 days	After full migration completes, you can reduce retention to more than 24 hours

Operational restrictions

During schema migration and full data migration: do not perform DDL operations (schema changes) on the source database. The task fails if DDL operations are detected.
During full and incremental migration: DTS temporarily disables constraint checks and foreign key cascade operations at the session level. Cascade update or delete operations on the source database may cause data inconsistency in the destination.
If you change the network type of the PolarDB-X 2.0 instance during migration, update the network connection settings in the DTS migration task as well.

Other limits

Evaluate the performance of the source and destination databases before you start data migration. Perform data migration during off-peak hours. Full data migration consumes read and write resources on the source and destination databases, which can increase the database load.
DTS attempts to resume failed migration tasks that are created within the last seven days. Therefore, before you switch your business to the destination instance, you must end or release the task, or use the revoke command to revoke the write permissions of the DTS account on the destination database. This prevents data in the destination from being overwritten after the task is automatically resumed.
If an instance fails, DTS helpdesk will try to recover the instance within 8 hours. During the recovery process, operations such as restarting the instance and adjusting parameters may be performed. When parameters are adjusted, only the parameters of the DTS instance are modified. The parameters of the database are not modified.
DTS periodically updates the dts_health_check.ha_health_check table in the source database to advance the binary log offset.

Permissions required

Grant the following permissions to the database accounts used by DTS:

Database	Schema migration	Full data migration	Incremental data migration
PolarDB-X 2.0	SELECT	SELECT	REPLICATION SLAVE, REPLICATION CLIENT, and SELECT on objects to be migrated
MaxCompute	—	Read and write	Read and write

For instructions on granting permissions to the PolarDB-X 2.0 account, see Account permission issues during data synchronization.

Billing

Migration type	Task configuration fee	Internet traffic fee
Schema migration + Full data migration	Free	Free when using Cloud Instance access. If you set the destination access method to Public IP Address, internet traffic fees apply. See Billing overview.
Incremental data migration	Charged. See Billing overview.	—

Create a migration task

The following steps use the DTS console. You can also use the DMS console — navigate to Data + AI > DTS (DTS) > Data Migration, then follow the same steps from step 2 onward.

Note

The actual DMS console layout may vary. See Simple mode and Customize the layout and style of the DMS console for more information.

Step 1: Open the Data Migration page

Log on to the DTS console.DTS consoleDMS console
In the left-side navigation pane, click Data Migration.
In the upper-left corner, select the region where the migration instance will reside.

Step 2: Create a task and configure source and destination databases

Warning

After you select the source and destination instances, read the Limits at the top of the page to confirm that the migration task can be created and run successfully.

Click Create Task.

Configure the source database:

Parameter	Description
Task Name	DTS generates a name automatically. Enter a descriptive name for easy identification. The name does not need to be unique.
Select Existing Connection	If the PolarDB-X 2.0 instance is already registered with DTS, select it from the drop-down list to pre-fill the parameters below. Otherwise, configure the parameters manually.
Database Type	Select PolarDB-X 2.0.
Access Method	Select Cloud Instance.
Instance Region	Select the region where the source PolarDB-X 2.0 instance resides.
Cross-account	Select No for migrations within the same Alibaba Cloud account.
Instance ID	Select the ID of the source PolarDB-X 2.0 instance.
Database Account	Enter the database account. See Permissions required for the required permissions.
Database Password	Enter the password for the database account.

Configure the destination database:

Parameter	Description
Database Type	Select MaxCompute.
Access Method	Select Cloud Instance.
Instance Region	Select the region where the destination MaxCompute project resides.
Project	Enter the name of the destination MaxCompute project.
AccessKey ID of Alibaba Cloud Account	Enter the AccessKey ID created in Prerequisites.
AccessKey Secret of Alibaba Cloud Account	Enter the AccessKey Secret.

Click Test Connectivity and Proceed.
Click OK to grant DTS the necessary permissions to access the destination MaxCompute project.

Step 3: Configure objects to migrate

On the Configure Objects page, configure the following parameters:

Parameter	Description
Migration Types	Select the migration types based on your strategy. See Choose a migration strategy. If you skip Schema Migration, create the destination tables in MaxCompute manually and enable object name mapping in Selected Objects.
Processing Mode for Existing Destination Tables	Precheck and Report Errors: fails the precheck if the destination already has tables with the same names. Ignore Errors and Proceed: skips the name conflict check. During full migration, DTS skips records with conflicting primary keys and keeps the existing destination record. During incremental migration, DTS overwrites the existing destination record. If source and destination schemas differ, the task may migrate only some columns or fail.
Additional Column Rule	DTS adds metadata columns to destination tables. Select New Rule or Old Rule based on your schema. Evaluate potential name conflicts before selecting.
Partition Definition of Incremental Data Table	Select a partition name if needed. See Partitions.
Capitalization of Object Names in Destination Instance	Controls the capitalization of database, table, and column names in the destination. See Specify the capitalization of object names in the destination instance.

In the Source Objects section, select the objects to migrate. Move them to the Selected Objects section.

To rename a single object in the destination, right-click the object in the Selected Objects section. See Map the name of a single object.
To rename multiple objects at a time, click Batch Edit in the upper-right corner of the Selected Objects section. See Map multiple object names at a time.

Step 4: Configure advanced settings

Click Next: Advanced Settings and configure the following:

Parameter	Description
Dedicated Cluster for Task Scheduling	By default, DTS schedules the task to the shared cluster. To improve stability, purchase and specify a dedicated cluster. See What is a DTS dedicated cluster.
Retry Time for Failed Connections	How long DTS retries a failed database connection before marking the task as failed. Valid values: 10–1,440 minutes. Default: 720 minutes. Set to at least 30 minutes. During retries, DTS instance fees continue to accrue.
Retry Time for Other Issues	How long DTS retries a failed DDL or DML operation. Valid values: 1–1,440 minutes. Default: 10 minutes. Set to at least 10 minutes. This value must be less than Retry Time for Failed Connections.
Enable Throttling for Full Data Migration	Limits the read and write load on the source and destination during full data migration. Configure QPS to the source database, RPS (rows per second), and migration speed (MB/s). Available only when Full Data Migration is selected.
Enable Throttling for Incremental Data Migration	Limits the load on the destination during incremental data migration. Configure RPS and migration speed (MB/s). Available only when Incremental Data Migration is selected.
Whether to delete SQL operations on heartbeat tables	Yesalert notification settings: DTS does not write to the heartbeat table. A latency indicator may appear for the DTS instance. No: DTS writes to the heartbeat table. This may affect physical backup and cloning of the source database.
Environment Tag	Optional. Tag to identify the environment, such as production or test.
Configure ETL	Whether to enable extract, transform, and load (ETL). Select Yes to enter data processing statements. See Configure ETL in a data migration or data synchronization task.
Monitoring and Alerting	Whether to configure alerts. If the task fails or migration latency exceeds the threshold, alert contacts receive notifications. Select Yes to configure the alert threshold and notification settings. See Configure monitoring and alerting.

Step 5: Run the precheck

Click Next: Save Task Settings and Precheck.

If the precheck passes, proceed to the next step.
If the precheck fails, click View Details, fix the reported issues, and rerun the precheck.
For alert items: either fix the issue or click Confirm Alert Details > Ignore > OK > Precheck Again if the alert is acceptable.

Step 6: Purchase the instance and start the task

After the precheck shows Success Rate at 100%, click Next: Purchase Instance.

On the Purchase Instance page, configure:

Parameter	Description
Resource Group	The resource group for the migration instance. Default: default resource group. See What is Resource Management?.
Instance Class	The instance class determines migration speed. See Instance classes of data migration instances.

Read and accept the Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start, then click OK in the confirmation dialog.

Monitor the migration task

Go to the Data Migration page to monitor progress:

Full data migration only: The task stops automatically when complete. Status shows Completed.
Incremental data migration included: The task runs continuously and never stops automatically. Status shows Running.

Before switching business workloads to MaxCompute, end or release the task, or use the revoke command to remove the DTS account's write permissions on the destination database. This is necessary because DTS attempts to resume failed migration tasks created within the last seven days, and you must prevent the task from automatically resuming and overwriting data in the destination.

Incremental log table structure

When DTS writes incremental data to MaxCompute, each incremental log table includes the following metadata fields in addition to the source table columns.

Note

Run set odps.sql.allow.fullscan=true; in MaxCompute before querying incremental log tables.

Field	Description
`record_id`	A unique, incrementally increasing ID for each log record. For UPDATE operations, DTS splits the change into two records (before and after the update) that share the same `record_id`.
`operation_flag`	The type of operation: `I` (INSERT), `D` (DELETE), or `U` (UPDATE).
`utc_timestamp`	The timestamp of the binary log entry, in UTC.
`before_flag`	Whether the row contains the column values before the update. Values: `Y` or `N`.
`after_flag`	Whether the row contains the column values after the update. Values: `Y` or `N`.

Data type mappings

For data type mappings between PolarDB-X 2.0 and MaxCompute, see Data type mappings for initial schema synchronization.

What's next

Map object names — rename tables or columns in the destination database.
Modify instance parameters — adjust DTS instance parameters after the task starts.
Billing overview — understand DTS pricing for incremental data migration.