Data Transmission Service:Migrate ApsaraDB for MongoDB (replica set architecture) to ApsaraDB for MongoDB (replica set or sharded cluster architecture)

Incremental migration: Oplog vs. Change Streams

Incremental migration captures changes from the source using either Oplog (recommended) or Change Streams.

Incremental migration does not capture databases created after the task starts.

Operations that cause task failure

Avoid the following actions during migration—they will cause the task to fail or result in data loss:

• Do not change database or collection schemas during schema migration or full data migration (including updates to array-type data).

• Do not write to the source during full-only migration. To maintain data consistency while the source stays live, run all three migration types together.

• For sharded cluster destinations: do not start the task without first purging orphaned documents. Documents with conflicting _id values cause task failure or data inconsistency.

• For sharded cluster destinations: add a sharding key to the source data that matches the destination sharding key before the task starts. After the task starts, INSERT operations must include the sharding key; UPDATE operations cannot change the sharding key.

Operations that may increase latency or cause inconsistency

These conditions do not stop the task but can degrade data quality:

• Collections with TTL indexes: TTL index conflicts between source and destination can cause latency and data inconsistency. See FAQ for details.

• Capped collections or unique indexes: These do not support concurrent replay during incremental migration—only single-threaded writes are used, which may increase latency.

• Transaction data: Transactions are not retained. Source transactions become individual documents in the destination.

• Primary key or unique key conflicts: When DTS writes to the destination and encounters a conflict, it skips the write and retains the existing destination data.

Source database requirements

General constraints

• Connecting via an SRV record is not supported.

• Migrate from a lower MongoDB version to a higher version. Migrating to a lower version may cause compatibility issues.

• The admin, config, and local databases cannot be migrated.

• If the source is MongoDB earlier than 3.6 and the destination is MongoDB 3.6 or later, field order in migrated documents may differ from the source due to differences in query plan execution. Field values remain consistent. If your business logic relies on text match queries against nested structures, assess the potential impact.

• If the source is MongoDB 5.0 or later and the destination is earlier than 5.0, migrating capped collections is not supported, because MongoDB 5.0 introduced support for explicit deletions and document size increases on update, which earlier versions do not support.

• Migrating time-series collections (introduced in MongoDB 5.0) is not supported.

• Make sure the destination has no documents with the same _id values as the source. Delete those documents from the destination before starting if any exist.

• DTS attempts to resume failed tasks automatically for up to 7 days. Before switching your application to the destination, end or release the task—or revoke write permissions from the DTS account—to prevent the source from overwriting destination data after an automatic resume.

• If the task fails, DTS technical support attempts recovery within 8 hours. During recovery, DTS may restart the task or adjust task parameters (not database parameters). For adjustable parameters, see Modify instance parameters.

Replica set destination requirements

• When Access Method is Express Connect, VPN Gateway, or Smart Access Gateway, Public IP Address, or Cloud Enterprise Network (CEN): set Domain Name or IP and Port Number to the primary node's address and port, or configure a high-availability connection address. See Create an instance with a high-availability MongoDB source or destination database.

• When Access Method is Self-managed Database on ECS: enter the primary node's port for Port Number.

Special cases for self-managed MongoDB sources

• A primary/secondary switchover during migration causes the task to fail.

• DTS calculates latency by comparing the timestamp of the last migrated record with the current time. If the source has been idle for a long time, the latency reading may be inaccurate. Perform an update on the source to refresh the latency display.

When migrating an entire database, create a heartbeat table that is updated every second to maintain accurate latency readings.

Step 1: Open the migration task list

Step 2: Create the task

Click Create Task.

Step 3: Configure source and destination databases

General settings:

Source database:

Destination database:

Step 4: Test connectivity

Click Test Connectivity and Proceed at the bottom of the page.

Add the CIDR blocks of DTS servers to the security settings of both the source and destination databases. See Add the IP address whitelist of DTS servers.

If the source or destination is a self-managed database (Access Method is not Alibaba Cloud Instance), also click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Step 5: Configure objects

On the Configure Objects page, configure the following:

If Schema Migration is not selected, make sure the destination already has the required databases and collections. If Incremental Data Migration is not selected, stop writes to the source during migration.

Click Next: Advanced Settings.

Step 6: Configure advanced settings

Click Next: Data Validation to configure data validation. For details, see Configure data validation.

Step 7: Save settings and run precheck

Click Next: Save Task Settings and Precheck.

To preview the API parameters for this configuration, hover over the button and click Preview OpenAPI parameters.

DTS runs a precheck before starting the task. The task starts only after passing.

If the precheck fails, click View Details next to the failed item, fix the issue, and rerun the precheck.

If a warning appears: for items that cannot be ignored, fix them and rerun. For ignorable items, click Confirm Alert Details > Ignore > OK > Precheck Again. Ignored warnings may lead to data inconsistency.

Step 8: Purchase and start the instance

1. When Success Rate reaches 100%, click Next: Purchase Instance.

2. On the Purchase page, select the instance class. For performance details, see Data migration link specifications.

   Parameter	Description
   Resource Group Settings	Select the resource group for the instance. See What is Resource Management?
   Instance Class	Select a specification based on your migration volume and performance requirements.

3. Read and select Data Transmission Service (Pay-as-you-go) Service Terms.

4. Click Buy and Start, then click OK in the confirmation dialog.

Track migration progress on the Data Migration Tasks list page.

Tasks without incremental migration stop automatically after full migration completes. The status changes to Completed.

Tasks with incremental migration continue running. The status remains Running until you manually stop the task.

Why does latency occur even when no data is being written to the source?

This is typically caused by TTL (Time To Live) indexes. When the TTL index on the source deletes an expired document, it generates a DELETE entry in the oplog. DTS synchronizes this DELETE to the destination. If the destination's TTL index already deleted the same document, DTS finds nothing to delete—the database engine returns an unexpected row count, triggering an exception that slows migration.

A related inconsistency can also occur: because TTL deletion is asynchronous, an expired document may still exist on the source at the moment the destination has already deleted it. This creates a gap between source and destination.

The MongoDB Oplog or ChangeStream records only the updated fields for an UPDATE operation. It does not record the full document before and after the update. Therefore, if an UPDATE operation cannot find the target data on the destination, DTS ignores the operation.

For example:

The document is now missing from the destination.

To resolve this, temporarily modify the TTL index expiration time on the destination during migration. For details, see Best practices for synchronizing/migrating collections with TTL indexes when MongoDB is the source.