Data Transmission Service:Migrate data from a self-managed MongoDB database (replica set architecture) to an ApsaraDB for MongoDB instance (replica set or sharded cluster architecture)

What incremental migration captures

Using oplog, DTS captures:

• CREATE COLLECTION and CREATE INDEX

• DROP DATABASE, DROP COLLECTION, and DROP INDEX

• RENAME COLLECTION

• Insert, update, and delete operations on documents (updates from $set only)

DTS does not migrate incremental data from databases created after the migration task starts.

Using change streams, DTS captures:

• DROP DATABASE and DROP COLLECTION

• RENAME COLLECTION

• Insert, update, and delete operations on documents (updates from $set only)

Source database

• Azure Cosmos DB for MongoDB clusters and Amazon DocumentDB elastic clusters: Only full data migration is supported.

• Collection requirements: Collections must have PRIMARY KEY or UNIQUE constraints with all fields unique; otherwise, duplicate records may appear in the destination.

• Collection count: When migrating by selecting individual collections, up to 1,000 collections per task. Exceeding this limit causes a request error. Migrate in batches or select entire databases instead.

• Document size: A single document cannot exceed 16 MB; exceeding this limit causes the task to fail.

• TTL indexes: Data inconsistency between source and destination may occur after migration.

• Schema changes: During schema migration and full data migration, do not modify schemas (including array type updates). Doing so causes task failure or data inconsistency.

• Writes during full-only migration: If you run only full data migration without incremental migration, do not write to the source database during migration. Writes cause data inconsistency.

• Primary/secondary switchover: If a primary/secondary switchover occurs on the source during migration, the task fails.

• Migration latency accuracy: DTS calculates migration latency by comparing the timestamp of the latest migrated data with the current source timestamp. If the source has no recent updates, the displayed latency may be inaccurate. Perform an update on the source to refresh the latency.

If you migrate an entire database, create a heartbeat table. DTS updates the heartbeat table every second, which keeps latency readings accurate.

Sharded cluster destination

• Orphaned documents: Delete orphaned documents before migration. _id conflicts between source and destination documents may cause data inconsistency or task failure.

• Shard keys: Add shard keys to data in the source before starting the task. If adding shard keys to the source is not possible, see Migrate data from a MongoDB instance without a sharding key to a MongoDB sharded cluster instance.

• Inserts during migration: Inserted data must contain shard keys.

• Updates during migration: Shard keys cannot be modified.

Replica set destination

• For connections over Express Connect, VPN Gateway, Smart Access Gateway, Public IP Address, or Cloud Enterprise Network (CEN): Set Domain Name or IP and Port Number to the primary node's IP address and port, or configure a high-availability endpoint. See Create a DTS task in which the source or destination database is a high-availability MongoDB database.

• For connections over Self-managed Database on ECS: Set Port Number to the port of the primary node.

General

• DTS cannot connect to a MongoDB database over an SRV endpoint.

• The destination MongoDB version must be the same as or later than the source version. Earlier destination versions may cause compatibility issues.

• DTS cannot migrate data from the admin, config, or local databases.

• Collections with a unique index or capped: true in the destination support only single-thread writes and do not support concurrent replay during incremental migration. This may increase migration latency.

• Transactions are not retained. During migration, transactions are converted into individual records.

• If a primary key or unique key conflict occurs, DTS skips the conflicting write operation and retains the existing data in the destination.

• DTS uses ROUND(COLUMN, PRECISION) to retrieve FLOAT and DOUBLE values. Default precision: 38 digits for FLOAT, 308 digits for DOUBLE. Verify that these defaults meet your requirements.

• Concurrent writes during full data migration cause fragmentation in destination collections. After full migration, destination storage space may exceed source storage space by 5–10%.

• DTS retries failed tasks for up to seven days. Before switching workloads to the destination, stop or release any failed tasks. Alternatively, revoke write permissions from the DTS account on the destination database to prevent the source from overwriting the destination after a task resumes.

• Query row counts on the destination using: db.$table_name.aggregate([{ $count:"myCount"}]).

• Make sure the destination database does not have the same _id values as the source. If conflicts exist, delete the conflicting data from the destination before starting the task.

• If a DTS instance fails, DTS helpdesk attempts to recover it within 8 hours. Recovery operations may include restarting the instance or adjusting parameters. See Modify instance parameters for parameters that may change.

Step 1: Go to the Data Migration page

Via the DTS console:

1. Log on to the DTS console.

2. In the left-side navigation pane, click Data Migration.

3. In the upper-left corner, select the region where the migration instance resides.

Via the DMS console:

The actual steps may vary based on the DMS console mode and layout. See Simple mode and Customize the layout and style of the DMS console.

1. Log on to the DMS console.

2. In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Data Migration.

3. From the drop-down list to the right of Data Migration Tasks, select the region where the migration instance resides.

Step 2: Create the task and configure source and destination databases

1. Click Create Task.

2. Configure the source and destination databases using the parameters in the following tables.

Source database parameters

Destination database parameters

Step 3: Test connectivity

Click Test Connectivity and Proceed, then click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Make sure the CIDR blocks of DTS servers are added to the allowlists of the source and destination databases. See Add DTS server IP addresses to a whitelist.

Step 4: Configure migration objects

On the Configure Objects page, configure the migration objects using the following parameters.

Parameter	Description
Migration Types	Select the migration types: Schema Migration, Full Data Migration, and optionally Incremental Data Migration. For zero-downtime migration, select all three. If you skip Schema Migration, create the target databases and collections in the destination before starting. If you skip Incremental Data Migration, do not write to the source during migration.
Processing Mode of Conflicting Tables	Precheck and Report Errors: fails the precheck if the destination contains collections with the same names as the source. Use object name mapping to rename migrated collections if existing collections cannot be deleted. Ignore Errors and Proceed: skips the precheck. DTS does not migrate records with the same primary keys as existing destination records. Data consistency is not guaranteed.
Capitalization of object names in destination instance	The capitalization policy for database and collection names in the destination. Default: DTS default policy. See Specify the capitalization of object names in the destination instance.
Source Objects	Select databases or collections to migrate. Click the icon to add them to Selected Objects.
Selected Objects	Right-click an object to rename it in the destination (object name mapping), set the incremental migration mode, or specify WHERE conditions for full migration data filtering. To remove an object, click it and then click the icon.

Step 5: Configure advanced settings

Click Next: Advanced Settings and configure the following parameters.

Step 6: Configure data verification (optional)

Click Next Step: Data Verification to set up a data verification task. See Configure a data verification task.

Step 7: Run the precheck

Click Next: Save Task Settings and Precheck.

To preview the API parameters for this task configuration, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

DTS runs a precheck before starting the migration. The task starts only after passing the precheck.

• If a check item fails, click View Details, resolve the issue, then click Precheck Again.

• If an alert is triggered for a check item that can be ignored, click Confirm Alert Details > Ignore > OK > Precheck Again. Ignoring alerts may result in data inconsistency.

Step 8: Purchase and start the instance

1. Wait for Success Rate to reach 100%, then click Next: Purchase Instance.

2. On the Purchase Instance page, configure the following parameters.

1. Read and select the checkbox to agree to Data Transmission Service (Pay-as-you-go) Service Terms.

2. Click Buy and Start, then click OK in the confirmation dialog box.