Data Transmission Service:Synchronize PolarDB-X 1.0 Data to Elasticsearch

Source database requirements

• Tables must have PRIMARY KEY or UNIQUE constraints. Tables with only UNIQUE constraints do not support schema synchronization — use PRIMARY KEY constraints where possible. Constraint fields must be unique; otherwise, duplicate data may appear in the destination.

• If you select tables as synchronization objects and need to configure column mappings, a single task supports a maximum of 1,000 tables. If the table count exceeds this limit, the task submission returns an error. In that case, split the tables across multiple tasks, or synchronize the entire database as a single object.

• The source PolarDB-X 1.0 instance must be version 5.2 or later.

• Binary logging is enabled by default on RDS MySQL instances attached to PolarDB-X 1.0. Verify that binlog_row_image is set to full — if it is not, the precheck will fail and the task will not start. To check the current value, run the following query on the RDS MySQL instance attached to PolarDB-X 1.0:

  SHOW GLOBAL VARIABLES WHERE Variable_name = 'binlog_row_image';

  Expected output when the parameter is correctly set:

  +------------------+-------+
  | Variable_name    | Value |
  +------------------+-------+
  | binlog_row_image | FULL  |
  +------------------+-------+

  If the value is not FULL, set it through the RDS console. See View instance parameters.

• Binary log retention requirements:

  • Incremental synchronization only: retain binary logs for at least 24 hours.

  • Full and incremental synchronization: retain binary logs for at least seven days. After full synchronization completes, you can reduce the retention period to more than 24 hours. If DTS cannot read the binary logs due to a short retention period, the task may fail or produce inconsistent data — issues caused by insufficient retention are not covered by the DTS Service-Level Agreement (SLA).

Operations to avoid during synchronization

• Do not change the network type of the source PolarDB-X 1.0 instance during synchronization. If you do, update the network connection settings of the synchronization task immediately.

• Do not scale out, scale in, synchronize hot spot tables, change shard keys, or make DDL changes on the source instance. These operations cause task failure or data inconsistency.

• Do not write new data to the source instance during full-only synchronization. To keep data consistent in real time, run schema synchronization, full data synchronization, and incremental data synchronization together.

• Do not run DDL operations that change database or table schemas during schema synchronization or full synchronization.

  During full synchronization, DTS queries the source database, which creates metadata locks that may block DDL operations.

Other limitations

• DTS does not synchronize INDEX, PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, or FOREIGN KEY objects.

• To add columns to a source table: first update the corresponding field mapping in the Elasticsearch instance, then run the DDL statement in the source database, and finally pause and restart the synchronization task.

• Synchronizing to Elasticsearch indexes that use parent-child relationships or Join field types is not supported and may cause task errors or query failures.

• Incremental synchronization consistency depends on the continuity of XA transactions in PolarDB-X 1.0. If XA transaction continuity is disrupted — for example, when modifying synchronization objects or during disaster recovery for the incremental data collection module — uncommitted XA transactions may be lost.

• Run synchronization during off-peak hours. Full synchronization consumes significant read and write resources on both the source and destination databases.

• DTS attempts to recover failed tasks automatically for up to seven days. Before switching traffic to the destination instance, stop or release the synchronization task, or use the revoke command to revoke DTS write access on the destination — this prevents automatic recovery from overwriting destination data.

• Development and test Elasticsearch instance specifications are not supported.

• If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust DTS task parameters (not database parameters). For parameters that may be changed, see Modify instance parameters.

DTS periodically updates the dts_health_check.ha_health_check table in the source database to advance the binary log offset.

Step 1: Open the data synchronization page

Open the data synchronization task list in the destination region using one of these methods:

Step 2: Configure the source and destination databases

Click Create Task, then configure the following settings on the task configuration page.

Task name

DTS generates a task name automatically. Specify a descriptive name for easy identification. Names do not need to be unique.

Source database

Destination database

After completing the configuration, click Test Connectivity and Proceed at the bottom of the page.

Add the CIDR blocks of the DTS servers to the security settings of both the source and destination databases. This can be done automatically or manually. See Add the IP address whitelist of DTS servers.

If the source or destination is a self-managed database (that is, Access Method is not Alibaba Cloud Instance), also click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Step 3: Configure the synchronization objects

On the Configure Objects page, set the following parameters.

Synchronization Types

DTS always enables Incremental Data Synchronization. By default, also select Schema Synchronization and Full Data Synchronization. After the precheck, DTS initializes the destination cluster with a full copy of the selected source objects as the baseline for incremental synchronization.

Processing Mode of Conflicting Tables

• Precheck and Report Errors: checks whether an index with the same name exists in the destination. If no conflict exists, the check passes. If a conflict is found, the precheck reports an error and the task does not start.

  If you cannot delete or rename the conflicting index, set an object name mapping for the synchronization object to avoid naming conflicts.

• Ignore Errors and Proceed: skips the conflict check for indexes that share a name with source objects.

  Warning

  This option may cause data inconsistency. If mapping structures match, records with the same primary key as the source are kept during initialization and overwritten during incremental synchronization. If mapping structures do not match, initialization may fail or only partial columns may synchronize.

Index Name

• Table Name: creates a destination index with the same name as the source table.

• Database Name_Table Name: creates a destination index named as <database name>_<table name>.

The selected index name format applies to all tables.

Capitalization of object names in destination instance

Configures the case-sensitivity policy for database, table, and column names. The DTS default policy is selected by default. See Case policy for destination object names.

Source Objects and Selected Objects

In the Source Objects box, select the objects to synchronize, then click to move them to the Selected Objects box.

Select objects at the table level. If you select an entire database, tables added or deleted in the source are not synchronized to the destination.

In the Selected Objects box:

• To rename a single object in the destination, right-click it. See Map a single object name.

• To rename multiple objects in bulk, click Batch Edit in the upper-right corner. See Map multiple object names in bulk.

• To filter rows using a WHERE clause, right-click the table and set the filter condition. See Set filter conditions.

Only underscores (_) are allowed as special characters in index and type names. Object name mapping may cause synchronization failures for other objects that depend on the renamed object.

Step 4: Configure advanced settings

Click Next: Advanced Settings and configure the following.

If multiple DTS instances share a source or destination database, the shortest configured Retry Time for Failed Connections among all instances applies to all of them.

DTS charges for task runtime during connection retries. Set the retry duration based on your business needs, or release the DTS instance promptly after releasing the source or destination.

If time-type data in the destination should not include time zone information, set the document type for that field in the destination Elasticsearch instance before starting synchronization.

You can also adjust the full synchronization throttle while the instance is running.

Step 5: Configure routing and document ID

Click Next: Configure Database and Table Fields to configure the _routing strategy and _id value for each table to be synchronized.

Step 6: Save and run the precheck

To view the OpenAPI parameters for this task, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters in the tooltip.

Click Next: Save Task Settings and Precheck to save the configuration and start the precheck.

DTS runs a precheck before starting any synchronization task. The task starts only if the precheck passes.

If the precheck fails, click View Details next to the failed item, resolve the issue as described, and rerun the precheck.

If the precheck returns warnings:

For non-ignorable warnings, click View Details, fix the issue, and run the precheck again.

For ignorable warnings, click Confirm Alert Details > Ignore > OK, then click Precheck Again to proceed. Ignoring warnings may lead to data inconsistency — proceed with caution.

Step 7: Purchase the instance

1. When the Success Rate reaches 100%, click Next: Purchase Instance.

2. On the Purchase page, select the billing method and link specifications.

   Category	Parameter	Description
   New Instance Class	Billing Method	Subscription — pay upfront for a set duration. Cost-effective for long-term, continuous tasks. Monthly options: 1–9 months. Yearly options: 1, 2, 3, or 5 years. Pay-as-you-go — billed hourly for actual usage. Suitable for short-term or test tasks.
   Resource Group Settings	—	The resource group for the instance. Defaults to default resource group. See What is Resource Management?
   Instance Class	—	DTS offers synchronization specifications at different performance levels. Select a specification based on your requirements. See Data synchronization link specifications.

3. Read and select Data Transmission Service (Pay-as-you-go) Service Terms.

4. Click Buy and Start, then click OK in the confirmation dialog box.

Monitor the task on the data synchronization page.