Data Transmission Service:Synchronize data from a MongoDB instance (without a sharding key) to a MongoDB instance (sharded cluster architecture)

Source database

Operations during synchronization

Do not perform the following operations during synchronization, as they may cause task failure or data inconsistency:

• Changing database or collection schemas (including array type updates) during schema synchronization or full data synchronization.

• Writing new data to the source when running full data synchronization only. To maintain real-time data consistency, select schema synchronization, full data synchronization, and incremental data synchronization together.

• Writing data to the destination from any source other than DTS.

Other limitations

• Newly added collections on the source cannot have shard key default values configured during synchronization.

• When the source is a non-sharded cluster, the task automatically enters the Configure Database and Table Fields stage to set shard key default values.

• ShardKey default values set in the Configure Database and Table Fields stage take effect only when the destination version is lower than 4.4. For destination instances running version 4.4 or later, DTS writes the original data as-is.

• Synchronize from a lower version to an equal or higher version to avoid compatibility issues.

• Synchronization of data in the admin and local databases is not supported.

• Collections with a unique index or with capped: true support only single-thread writes and cannot use concurrent replay during incremental synchronization. This may increase synchronization latency.

• Transaction information is not retained. Transactions in the source are converted to individual records in the destination.

• Before performing data synchronization, evaluate the performance of the source and destination databases. It is also recommended to perform data synchronization during off-peak business hours to reduce the load on the source and destination databases.

• Because full data synchronization performs INSERT operations concurrently, it causes fragmentation in destination collections. After full synchronization completes, the destination storage space will be 5–10% larger than the source.

• DTS automatically attempts to recover failed tasks for up to 7 days. Before switching traffic to the destination, end or release the DTS task, or revoke DTS write access to the destination — otherwise the auto-recovery may overwrite destination data.

• Query the document count in the destination MongoDB using: db.$table_name.aggregate([{ $count:"myCount"}]).

• Make sure the destination has no documents sharing the same primary key (_id) with the source. If duplicates exist, delete the conflicting documents from the destination before starting the task.

• If a DTS task fails, DTS support will attempt to restore it within 8 hours and may restart the task or modify task parameters (not database parameters). For parameters that may be modified, see Modify instance parameters.

Self-managed MongoDB

If the source is a self-managed MongoDB instance:

• A primary-secondary switch during synchronization will cause the task to fail.

• If the source database has not had any write operations for an extended period, the displayed synchronization latency may be inaccurate. Perform an update operation on the source to refresh the latency. Alternatively, if the synchronization object is the entire database, configure a heartbeat that writes data to the source every second.

Choose between oplog and change streams

DTS supports two methods for capturing incremental changes. Use oplog unless your source requires change streams.

When DTS synchronizes incremental data of a file, only the $set command can be run synchronously. DTS does not synchronize incremental data from databases created after the task starts.

Step 1: Go to the Data Synchronization page

Use one of the following methods:

Step 2: Create the task and configure connections

1. Click Create Task.

2. If prompted, click New Configuration Page in the upper-right corner.

   Skip this step if Back to Previous Version is displayed instead. The new configuration page is recommended.

3. Configure the source and destination databases using the following parameters.

Task settings

Source database

Destination database

1. Click Test Connectivity and Proceed.

   DTS server CIDR blocks must be added to the security settings of the source and destination databases. For details, see Add the CIDR blocks of DTS servers. If the source or destination is a self-managed database with an access method other than Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Step 3: Configure synchronization objects

1. In the Configure Objects step, set the following parameters.

1. Click Next: Advanced Settings.

Step 4: Configure advanced settings

Step 5: Configure data verification

Click Next Step: Data Verification to configure data verification. For more information about how to use the data verification feature, see Configure a data verification task.

Step 6: Set shard key default values

This step applies only when the source is a non-sharded cluster and the destination is a sharded cluster. DTS automatically enters the Configure Database and Table Fields stage. Shard key default values take effect only when the destination version is lower than 4.4. For destination instances running version 4.4 or later, DTS writes the original data as-is.

For each collection in the synchronization scope that requires a shard key:

1. Click Set Default Value in the corresponding row.

   If Number of Shard Keys shows 0 for a collection, that collection has no shard key in the destination and does not need a default value.

2. Select a Shard key default value type. Supported types: string and int.

3. Enter the Default Value for the shard key.

   Important

   Assign default values to all shard keys for all objects in the synchronization scope. Missing values trigger an alert during the precheck and may cause the task to fail.

Step 7: Save the task and run a precheck

Click Next: Save Task Settings and Precheck.

To preview the API parameters for this task configuration, hover over the button and click Preview OpenAPI parameters before proceeding.

DTS runs a precheck before starting the task. If the precheck fails:

• For failed items: click View Details, fix the underlying issue, then rerun the precheck.

• For alert items that can be ignored: click Confirm Alert Details, then click Ignore in the View Details dialog box, confirm with OK, and click Precheck Again. Ignoring alerts may result in data inconsistency.

Step 8: Purchase the instance and start the task

1. Wait until the Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the buy page, configure the following parameters.

1. Read and select Data Transmission Service (Pay-as-you-go) Service Terms.

2. Click Buy and Start, then click OK in the confirmation dialog box.

The task appears in the task list. Monitor its progress from there.