Data Transmission Service:Migrate MongoDB (without shard keys) to a MongoDB sharded cluster

Shard key behavior depends on the destination MongoDB version

How DTS handles default shard key values depends on the destination MongoDB version:

• Destination MongoDB earlier than 4.4: The default shard key value you set in the Configure Database and Table Fields step takes effect. DTS fills the original data with this default value and writes it to the destination.

• Destination MongoDB 4.4 or later: The default shard key value does not take effect. DTS writes the original data to the destination as-is.

Evaluate which behavior applies to your destination instance before proceeding.

Data sharding and pre-sharding

If your destination requires sharding, create the databases and collections that need sharding in the destination instance before migration. Configure data sharding, enable the balancer, and perform pre-sharding. This prevents all data from landing on a single shard and avoids data skew after migration.

For details, see Configure data sharding to maximize shard performance and How do I handle uneven data distribution in a MongoDB sharded cluster?.

Version compatibility

Keep the source and destination MongoDB versions the same, or migrate from an earlier version to a later version. Migrating from a later version to an earlier version may cause compatibility issues.

Incremental migration with oplog

DTS captures incremental data through the following operations:

• CREATE COLLECTION, CREATE INDEX

• DROP DATABASE, DROP COLLECTION, DROP INDEX

• RENAME COLLECTION

• Insert, update, and delete operations on documents

For updated documents, DTS migrates only updates from $set. Incremental migration does not capture data from databases created after the task starts. Use oplog for incremental migration whenever possible -- it provides faster log pulling and lower latency than change streams.

Incremental migration with change streams

DTS captures incremental data through the following operations:

• DROP DATABASE, DROP COLLECTION

• RENAME COLLECTION

• Insert, update, and delete operations on documents

For updated documents, DTS migrates only updates from $set.

Source database

• Bandwidth: The source server must have enough outbound bandwidth. Insufficient bandwidth slows migration.

• Primary key or unique constraint required: Collections to migrate must have a primary key or UNIQUE constraint with unique fields. Otherwise, duplicate data may appear in the destination.

• Collection limit per task: When migrating at the collection level with object editing (such as name mapping), a single task supports up to 1,000 collections. If you exceed this limit, split the collections into multiple tasks or migrate the entire database.

• Document size limit: Each document in the source must be 16 MB or smaller. Larger documents cause the task to fail.

• Mongos node limit: If the source is a sharded cluster, it must have 10 or fewer mongos nodes.

• Self-managed sharded cluster access methods: Only Public IP Address, Express Connect, VPN Gateway, or Smart Access Gateway, and Cloud Enterprise Network (CEN) are supported.

• MongoDB 8.0+ with oplog: If the source is a self-managed sharded cluster running MongoDB 8.0 or later and Migration Method is Oplog, grant the directShardOperations permission to the shard account: Replace username with the shard account used for the migration task.

    db.adminCommand({ grantRolesToUser: "username", roles: [{ role: "directShardOperations", db: "admin"}]})

• Azure Cosmos DB / Amazon DocumentDB elastic clusters: Only full data migration is supported.

• Oplog retention for incremental migration: The oplog feature must be enabled with at least seven days of operation log retention. Alternatively, enable change streams with a seven-day subscription window. If DTS cannot access the logs, the task may fail, and data inconsistency or loss may occur. The DTS SLA does not cover these issues.

• Change streams version requirement: Change streams require MongoDB V4.0 or later.

• Inelastic Amazon DocumentDB clusters: Enable change streams, set Migration Method to ChangeStream, and set Architecture to Sharded Cluster.

• TTL indexes: TTL indexes in the source may cause data inconsistency between source and destination after migration.

• Orphaned documents: Make sure no orphaned documents exist in a source sharded cluster. Otherwise, data inconsistency or task failure may occur. See Orphaned Documents and How do I clean up orphaned documents in a MongoDB sharded cluster?.

• Balancer activity: If the source sharded cluster balancer is actively rebalancing data, instance latency may increase.

Restrictions during migration

• Schema migration and full data migration: Do not perform schema changes on databases or collections, including updates to array types. Schema changes may cause the task to fail or result in data inconsistency.

• Sharded cluster sources: Do not run shardCollection, reshardCollection, unshardCollection, moveCollection, or movePrimary to change data distribution while the migration instance is running. Otherwise, data inconsistency may occur.

• Full migration only (no incremental): Do not write new data to the source. To maintain real-time consistency, select all three migration types: Schema Migration, Full Data Migration, and Incremental Data Migration.

General limitations

• New collections during migration: Default shard key values cannot be set for collections added to the source after the task starts.

• SRV addresses: DTS does not support connecting to MongoDB through an SRV address.

• Non-sharded source to sharded destination: When the source is a non-sharded cluster and the destination is an Alibaba Cloud sharded cluster, the task proceeds to the Configure Database and Table Fields step.

• Excluded databases: DTS cannot migrate data from the admin, config, or local databases.

• Transactions: Transaction information is not retained. Transactions in the source are converted to individual records in the destination.

• Primary key / unique key conflicts: When a conflict occurs, DTS skips the conflicting write and retains the existing data in the destination.

• Field order (source < 3.6, destination >= 3.6): If the source runs MongoDB earlier than 3.6 and the destination runs 3.6 or later, field order may be inconsistent after migration due to differences in database engine execution plans. Evaluate the impact if your business logic involves match queries on nested structures.

• Storage overhead from concurrent writes: Full migration runs concurrent INSERT operations, which cause collection fragmentation. The destination occupies 5% to 10% more storage than the source.

• Automatic task retry: DTS retries failed tasks within the last seven days. Before switching workloads to the destination, stop or release the task, or revoke write permissions on the destination account. This prevents automatic resumption from overwriting destination data.

• Unique index or capped collections: Collections with a unique index or the capped attribute set to true support only single-thread writes during incremental migration (no concurrent replay), which may increase latency.

• Document count query: To query the number of documents in the destination, use: db.$table_name.aggregate([{ $count:"myCount"}]).

• Duplicate primary keys (_id): Make sure the destination does not have documents with the same _id values as the source. Duplicate _id values cause data loss. Clear conflicting documents in the destination before migration if doing so does not affect your business.

• DTS task failure restoration: If a task fails, DTS technical support attempts restoration within 8 hours. During restoration, the task may be restarted and DTS task parameters may be modified. Database parameters are not modified. Modified parameters include but are not limited to those described in Modify instance parameters.

• Sharded collection compliance: After switching your business to the destination, all business operations must comply with the requirements for sharded collections in that MongoDB database.

Self-managed MongoDB sources

• Failover during migration: If a primary/secondary failover occurs during migration, the task fails.

• Latency calculation: DTS calculates latency by comparing the timestamp of the last migrated entry with the current time. If the source has not been updated for a long time, the displayed latency may be inaccurate. Run an update on the source to refresh latency.

• Heartbeat for whole-database migration: Create a heartbeat collection that periodically updates or writes data (for example, every second) to keep latency accurate.

Step 1: Open the data migration page

Use one of the following methods to navigate to the Data Migration page.

DTS console:

1. Log on to the DTS console.

2. In the left-side navigation pane, click Data Migration.

3. In the upper-left corner, select the region where the data migration instance resides.

DMS console:

The actual operation may vary based on the mode and layout of the DMS console. For more information, see Simple mode and Customize the layout and style of the DMS console.

1. Log on to the DMS console.

2. In the top navigation bar, hover over Data + AI > DTS (DTS) > Data Migration.

3. From the drop-down list next to Data Migration Tasks, select the region where the data migration instance resides.

Step 2: Create a migration task

Click Create Task to open the task configuration page.

Step 3: Configure the source and destination databases

Configure the following parameters for the source and destination databases.

Task settings

Source database

Destination database

Step 4: Test connectivity

Click Test Connectivity and Proceed at the bottom of the page.

Make sure the CIDR blocks of DTS servers are added to the security settings of both the source and destination databases. For details, see Add the CIDR blocks of DTS servers. If the source or destination is a self-managed database and its Access Method is not Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Step 5: Configure migration objects

On the Configure Objects page, configure the following settings.

Step 6: Configure advanced settings

Click Next: Advanced Settings and configure the following parameters.

Step 7: Configure data verification

Click Next Step: Data Verification and configure the data verification task. For details, see Configure a data verification task.

Step 8: Set default shard key values

Click Next: Configure Database and Table Fields to set the default shard key value for collections without shard keys.

1. In the row of the destination collection, click Set Default Value.

   If the Number of Shard Keys for a collection is 0, the collection has no shard key and does not need a default value.

2. Select a Shard key default value type.

   Only string and int types are supported.

3. Enter the Default Value for the shard key.

Step 9: Run the precheck

Click Next: Save Task Settings and Precheck.

To preview the API parameters for this task, hover over the button and click Preview OpenAPI parameters.

DTS runs a precheck before starting the migration. The task starts only after the precheck passes.

• If the precheck fails, click View Details next to each failed item, troubleshoot the issue, and run the precheck again.

• If a precheck item triggers a warning:

  • If the warning cannot be ignored, click View Details, fix the issue, and rerun the precheck.

  • If the warning can be ignored, click Confirm Alert Details > Ignore > OK, then click Precheck Again. Ignoring warnings may cause data inconsistency.

Step 10: Purchase the instance and start migration

1. Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the Purchase Instance page, configure the instance:

   Parameter	Description
   Resource Group	The resource group for the migration instance. Default: default resource group. See What is Resource Management?.
   Instance Class	Select a class based on your required migration speed. See Instance classes of data migration instances.

3. Read and accept the Data Transmission Service (Pay-as-you-go) Service Terms.

4. Click Buy and Start, then click OK in the confirmation dialog.

The task appears on the Data Migration page where you can monitor progress.

• If the task does not include incremental migration, it stops automatically and the Status shows Completed.

• If the task includes incremental migration, it runs continuously and the Status shows Running. The task does not stop automatically.