Data Transmission Service:Synchronize data from ApsaraDB for MongoDB to Lindorm

Source database limitations

• The source server must have sufficient outbound bandwidth, or synchronization speed is affected.

• Collections to be synchronized must have PRIMARY KEY or UNIQUE constraints, and all field values must be unique. Otherwise, duplicate records may appear in the destination.

• DTS cannot connect to a MongoDB instance over a SRV endpoint.

• The source cannot be an Azure Cosmos DB for MongoDB cluster or an Amazon DocumentDB elastic cluster.

• A single data entry cannot exceed 16 MB.

• If the source is a sharded cluster instance:

  • The _id field in each collection must be unique. Otherwise, data inconsistency may occur.

  • The number of Mongos nodes cannot exceed 10.

  • Do not run the following commands during synchronization: shardCollection, reshardCollection, unshardCollection, moveCollection, or movePrimary. These commands change data distribution and can cause data inconsistency.

  • Make sure the source instance has no orphaned documents before starting the task. See the MongoDB documentation and the DTS FAQ for how to remove them.

  • If the balancer is active on the source, it may cause latency.

• If the source contains TTL indexes, data inconsistency may occur after synchronization.

• The oplog must be enabled and retain at least 7 days of data. Alternatively, change streams must be enabled and cover the last 7 days. If neither condition is met, DTS may fail to obtain incremental changes, which can result in data loss or inconsistency — not covered by the DTS service level agreement (SLA).

  • Use the oplog to record data changes when possible.

  • Change streams require MongoDB 4.0 or later and do not support two-way synchronization.

  • For non-elastic Amazon DocumentDB clusters, use change streams and set Migration Method to ChangeStream and Architecture to Sharded Cluster.

• During full data synchronization, do not modify database or collection schemas, or data of the ARRAY type. Do not write to the source database if running full synchronization only.

• If you select collections as the objects to synchronize and need to rename them in the destination, a single task supports up to 1,000 collections. For more than 1,000 collections, configure multiple tasks or select the entire database as the synchronization object.

Destination and task limitations

• Only tasks within the Germany (Frankfurt) region are supported.

• DTS cannot synchronize data from the admin, config, or local databases.

• The destination Lindorm instance cannot have collections with _id or _value columns.

• Transactions are not retained. DTS converts each transaction to individual records at the destination.

• The data written to Lindorm must meet the requirements in Limits on data requests.

• For UPDATE and DELETE operations in incremental synchronization:

  • If the wide table is created using Lindorm SQL, add a non-primary key column named _mongo_id_ when creating the table. The column's data type must match the _id column type in the source. Create a secondary index on this column.

  • If the wide table is created using the Apache HBase API, add a non-primary key column named _mongo_id_ with column family f. The column's data type must match the _id column type in the source. Create a secondary index on this column. If you plan to add extra columns and use the extract, transform, and load (ETL) feature, make sure the Lindorm instance has no duplicate data.

• DTS uses ROUND(COLUMN,PRECISION) to retrieve values from FLOAT and DOUBLE columns. The default precision is 38 digits for FLOAT and 308 digits for DOUBLE. Verify that these precision settings meet your requirements.

• DTS attempts to resume failed tasks for up to 7 days. Before switching workloads to the destination, stop or release any failed tasks, or run REVOKE to remove DTS write permissions on the destination. Otherwise, resumed tasks will overwrite destination data with source data.

• Run the task during off-peak hours. Full data synchronization reads and writes both source and destination databases, which increases server load and causes temporary fragmentation in destination collections.

• Synchronization latency is calculated from the latest synchronized data timestamp versus the current source timestamp. If the source has no writes for an extended period, this metric may be inaccurate. Perform a write on the source to refresh the latency value.

• If a DTS task fails, technical support attempts to restore it within 8 hours. The task may be restarted and certain task parameters may be modified during restoration. Database parameters are not modified.

Step 1: Open the data synchronization page

Use one of the following methods.

Step 2: Configure source and destination databases

Click Create Task, then configure the parameters.

Task settings:

Source database:

Destination database:

Step 3: Test connectivity

Click Test Connectivity and Proceed.

DTS automatically adds its server CIDR blocks to the security settings of the source and destination databases. If you are using a self-managed database with an access method other than Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box to confirm connectivity. For details, see Add the CIDR blocks of DTS servers.

Step 4: Configure objects to synchronize

In the Configure Objects step, set the following parameters.

Parameter	Description
Synchronization Types	Incremental Data Synchronization is selected by default. You can also select Full Data Synchronization only. Schema synchronization is not available.
Processing Mode of Conflicting Tables	Keep the default setting.
Capitalization of Object Names in Destination Instance	Controls the capitalization of database and collection names at the destination. Default is DTS default policy. See Specify the capitalization of object names in the destination instance.
Source Objects	Select one or more objects, then click to move them to Selected Objects. You can select individual collections.
Selected Objects	Configure column mappings for each collection. Columns not added to the destination wide table are not synchronized.

Map MongoDB fields to Lindorm columns:

When you add a collection to Selected Objects, DTS pre-populates a bson_value() expression for each field. Follow these steps to finalize the column mapping.

1. Edit the database name (optional): Right-click the database in Selected Objects, enter the target schema name in the Edit Schema dialog, and select the DML operations to synchronize. Click OK.

   

2. Edit the collection name (optional): Right-click the collection in Selected Objects, enter the target table name in the Edit Table Name dialog, and optionally specify filter conditions or select DML operations. Click OK.

   

3. Configure column mappings: DTS auto-generates a bson_value() expression for each row. Check each row and specify Column Name, Type, Length, and Precision. In each bson_value() expression, the string in "" is the field name in the source MongoDB document. For example, bson_value("age") maps the age field. To remove a field from synchronization, click next to that row.

   The expression meets the requirements

   • Set Column Name:

     • For SQL-created tables: use the destination column name.

     • For Apache HBase API tables: use ROW for the primary key column, and Column family:Column name (for example, person:name) for non-primary key columns. Create column mappings first if needed — see Add column mappings for an Apache HBase API table.

   • Select the Type for each column. Make sure it is compatible with the source data type.

   • Optionally set Length and Precision.

   • Repeat for all columns.

   The expression does not meet the requirements

   1. Click next to the row, then click + New Column.

   2. Set Column Name, Type, Length, and Precision.

   3. Enter the bson_value() expression in Assign Value. See Value assignment example for reference. > Important: Assign bson_value("_id") to the primary key column. For nested fields, specify the full hierarchical path in bson_value(). For example, use bson_value("person","name") instead of bson_value("person"). Using only the parent field causes data loss or task failure for incremental operations.

   4. Repeat for all columns.

4. Click OK.

Step 5: Configure advanced settings

Click Next: Advanced Settings and configure the following parameters.

Step 6: Run the precheck

Click Next: Save Task Settings and Precheck.

To view the OpenAPI parameters for this task configuration, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters before proceeding.

DTS runs a precheck before starting the task. If the precheck fails:

• Click View Details next to the failed item, resolve the issue, and click Precheck Again.

• If an item shows an alert that can be ignored: click Confirm Alert Details, then in the View Details dialog click Ignore > OK, then click Precheck Again. Note that ignoring alerts may result in data inconsistency.

Step 7: Purchase an instance

1. Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the buy page, configure the following parameters.

1. Read and accept Data Transmission Service (Pay-as-you-go) Service Terms.

2. Click Buy and Start, then click OK in the confirmation dialog.

The task appears in the task list. You can monitor its progress there.