Data Transmission Service:Migrate data from PolarDB-X 1.0 to Elasticsearch

Before you start

Review these limitations before configuring your migration task:

• Tables to be migrated must have primary keys or UNIQUE constraints with unique field values. Otherwise, duplicate data may appear in the destination.

• When migrating at the table level with column mapping, a single task supports a maximum of 1,000 tables. If you exceed this limit, split the tables into multiple batches and create a separate task for each batch, or configure the task to migrate an entire database.

• PolarDB-X 1.0 read-only instances do not support data migration.

• DTS does not support migrating INDEX, PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, or FK objects.

• You cannot migrate data to an Elasticsearch index that has a parent-child relationship or a Join field type mapping. Attempting this may cause the task to fail or break data queries in the destination.

• Timestamp fields with a value of 0 in the source become null in the destination.

For incremental migration, binary logs must meet these requirements:

• Binary logging must be enabled, and binlog_row_image must be set to full. If not, the precheck fails and the task cannot start.

• Binary log retention period: If the retention period is shorter than required, DTS may fail to retrieve binary logs. In extreme cases, this causes data inconsistency or data loss. Issues caused by insufficient log retention are not covered by the Service-Level Agreement (SLA).

  • Incremental migration only: at least 24 hours

  • Full migration + incremental migration: at least 7 days. After full migration completes, you can reduce this to 24 hours or more.

During migration

Avoid the following operations while the migration task is running:

• On the source database: Do not perform scale-out, scale-in, hot spot table migration, shard key changes, or DDL operations. These cause the task to fail.

  During full migration, DTS queries the source database, which creates a metadata lock that may block DDL operations on the source.

• On the source database: DTS temporarily disables foreign key constraint checks and cascade operations at the session level. If you run cascade update or delete operations during migration, data inconsistency may occur.

• On the source database: If you change the network type of the PolarDB-X 1.0 instance during migration, update the network connectivity settings in the migration task as well.

• Schema changes: To add columns to tables in the source database, first update the corresponding mapping in the Elasticsearch instance, then run the DDL statement in the source, then pause and restart the migration task.

XA transaction consistency: DTS ensures data consistency for incremental migration based on the continuity of XA transactions in PolarDB-X 1.0. If XA transaction continuity is interrupted — for example, during a disaster recovery event for the incremental data collection module — uncommitted XA transactions may be lost.

After migration

• Before switching production traffic to the destination, end or release the migration task. Alternatively, revoke write permissions for the DTS account on the destination using the REVOKE command. This prevents automatic task recovery from overwriting destination data with source data.

• DTS attempts to auto-recover failed tasks for up to 7 days. DTS support staff will attempt to restore failed tasks within 8 hours. During restoration, they may restart the task or adjust task parameters (not database parameters). Parameters that may be adjusted are listed in Modify instance parameters.

Other notes

DTS periodically updates the dts_health_check.ha_health_check table in the source database to advance the binary log offset.

Step 1: Open the migration task list

Navigate to the Data Migration task list using one of the following methods.

Step 2: Create a task and configure connections

1. Click Create Task.

2. Configure the source database (PolarDB-X 1.0):

   Parameter	Description
   Task Name	DTS generates a name automatically. Specify a descriptive name for easy identification. The name does not need to be unique.
   Select Existing Connection	Select a previously added database instance from the drop-down list to auto-fill the connection details. If no saved instance exists, configure the parameters below manually.
   Database Type	Select PolarDB-X 1.0.
   Connection Type	Select Cloud instance.
   Instance Region	Select the region where the source instance resides.
   Cross Alibaba Cloud account?	Select Not cross-account if the source and destination are under the same Alibaba Cloud account.
   RDS instance ID	Select the source PolarDB-X 1.0 instance ID.
   Database Account	Enter the database account. See Required database account permissions.
   Database Password	Enter the password for the database account.

3. Configure the destination database (Elasticsearch):

   Parameter	Description
   Select Existing Connection	Select a previously added database instance from the drop-down list to auto-fill the connection details. If no saved instance exists, configure the parameters below manually.
   Database Type	Select Elasticsearch.
   Connection Type	Select Cloud instance.
   Instance Region	Select the region where the destination instance resides.
   Type	Select Cluster or Serverless.
   Instance ID	Select the destination Elasticsearch instance ID.
   Database Account	Enter the logon name specified when creating the Elasticsearch instance. The default is elastic.
   Database Password	Enter the password for the database account.
   Encryption	Select HTTP or HTTPS.

4. Click Test Connectivity and Proceed.

   Make sure DTS server IP addresses are added to the security settings (whitelist) of both the source and destination databases. See Add DTS server IP addresses to a whitelist. If the source or destination uses a public IP address (that is, Access Method is not Alibaba Cloud Instance), also click Test Connectivity in the CIDR Blocks of DTS Servers dialog.

Step 3: Select migration objects

On the Configure Objects page:

1. Set the migration options:

   Parameter	Description
   Synchronization Type	Select Schema Migration and Full Data Migration for a one-time migration. Add Incremental Data Migration to keep source and destination in sync during the migration period. See Choose a migration strategy. Important If you do not select Schema Migration, you must ensure that a database and tables to receive the data already exist in the destination database.
   Processing Mode for Existing Destination Tables	Precheck and Report Errors: Checks for same-name tables in the destination before starting. The task fails the precheck if conflicts exist. To resolve conflicts, rename tables in the destination — see Object name mapping. Ignore Errors and Proceed: Skips the check. Warning This may cause data inconsistency. During full migration, DTS keeps records already in the destination. During incremental migration, source records overwrite destination records with the same primary key.
   Index name	Table name: Uses the table name as the Elasticsearch index name. Database name_Table name: Creates an index name by concatenating the database name, an underscore (_), and the table name. The chosen format applies to all migrated tables.
   Case Policy for Destination Object Names	Set the case sensitivity policy for object names in the destination. The default is DTS default policy. See Case sensitivity policy for destination database object names.

2. In the Source Objects box, click the objects to migrate, then click the right arrow to move them to the Selected Objects box.

   Select individual tables rather than entire databases. Database-level selection does not capture future table additions or deletions.

3. In the Selected Objects box, optionally:

   • Right-click an object to rename it. See Individual table column mapping.

   • Click Batch Edit to rename multiple objects at once. See Map multiple object names at a time.

   • Right-click an object to set a WHERE clause filter. See Set filter conditions.

   • Right-click an object to select specific SQL operations to migrate at the database or table level.

   Only underscores (_) are allowed as special characters in index and type names. Using object name mapping may cause migration failures for dependent objects.

Step 4: Configure advanced settings

Click Next: Advanced Settings and configure the following parameters:

Step 5: Configure routing and document ID

Click Next: Configure database and table fields and set the routing and document ID for each table:

Step 6: Save the task and run a precheck

1. Click Next: Save Task Settings and Precheck. To preview the API parameters for this task, hover over the button and click Preview OpenAPI parameters.

2. Wait for the precheck to complete.

   DTS runs a precheck before starting the migration. The task starts only after it passes the precheck. If the precheck fails, click View Details next to the failed item, fix the issue, and rerun the precheck. For warnings on items that can be ignored, click Confirm Alert Details > Ignore > OK > Precheck Again to proceed. Ignoring warnings may cause data inconsistency.

Step 7: Purchase the migration instance

1. After the Success Rate reaches 100%, click Next: Purchase Instance.

2. On the Purchase page, select the instance class:

   Parameter	Description
   Resource Group Settings	Select the resource group. The default is the default resource group. See What is Resource Management?
   Instance Class	The link specification determines migration speed. Select a class based on your data volume and performance requirements. See Data migration link specifications.

3. Read and accept the Data Transmission Service (Pay-as-you-go) Service Terms.

4. Click Buy and Start, then click OK in the confirmation dialog.