Data Transmission Service:Migrate MongoDB (without shard keys) to a MongoDB sharded cluster

Source database limits

• The source server must have sufficient outbound bandwidth. Insufficient bandwidth reduces migration speed.

• Collections to be migrated must have a primary key or UNIQUE constraint with unique fields. Otherwise, duplicate data may appear in the destination.

• When the migration granularity is collection with name mapping, a single task supports up to 1,000 collections. Exceeding this limit triggers a request error. In that case, split the collections across multiple tasks, or migrate entire databases instead.

• A single document in the source cannot exceed 16 MB.

• If the source is a sharded cluster, the number of source Mongos nodes cannot exceed 10.

• If the source is an Azure Cosmos DB for MongoDB cluster or an Amazon DocumentDB elastic cluster, only full data migration is supported.

• For incremental data migration, one of the following must be met:

  • The oplog feature is enabled and operation logs are retained for at least 7 days.

  • Change streams are enabled and DTS can subscribe to changes from the last 7 days.

  Important

  Use the oplog feature to get data changes from the source when possible. Change streams require the source to run MongoDB V4.0 or later. If the source is a non-elastic Amazon DocumentDB cluster, change streams are required — set Migration Method to ChangeStream and Architecture to Sharded Cluster when configuring the task.

• During schema migration and full data migration, do not change the schema of any database or collection (including array type updates). Doing so causes task failure or data inconsistency.

• If you run only full data migration (without incremental), do not write new data to the source during migration.

• Collections with time to live (TTL) indexes cannot be migrated. If such collections exist in the source, data inconsistency may occur after migration.

• If the source is a sharded cluster, make sure there are no orphaned documents. Orphaned documents can cause data inconsistency or task failure. See Orphaned document and How to clean orphaned documents.

• If the source is a sharded cluster with an active Balancer (data balancing in progress), migration latency may increase.

Other limits

• New collections added to the source after the task starts do not support default shard key values.

• If the destination MongoDB (sharded cluster) version is lower than 4.4, the default ShardKey value takes effect. DTS fills original data with the specified default value and writes it to the destination.

• If the destination MongoDB (sharded cluster) version is 4.4 or later, the default ShardKey value does not take effect. DTS writes original data to the destination as-is.

• Keep the source and destination MongoDB versions consistent, or migrate from a lower version to a higher version. Migrating from a higher version to a lower version may cause compatibility issues.

• The admin and local databases cannot be migrated.

• Transaction information is not retained. Transactions are converted to individual records in the destination.

• Run the migration during off-peak hours. During full data migration, DTS uses read and write resources on both the source and destination, which increases database load.

• Full data migration uses concurrent INSERT operations, which causes fragmentation in destination collections. After full migration, the destination storage space will be larger than the source.

• Because DTS writes data concurrently, the destination storage space is 5–10% larger than the source.

• DTS attempts to resume failed tasks from the last 7 days. Before switching your workloads to the destination, stop or release the task, or revoke the write permissions from the DTS accounts on the destination — otherwise the task may resume automatically and overwrite destination data.

• If a destination collection has a unique index or the capped attribute is true, the collection supports only single-thread writes and does not support concurrent replay during incremental migration. This may increase latency.

• To query the document count in the destination, use: db.$table_name.aggregate([{ $count: "myCount" }])

• Make sure the destination does not already contain documents with the same primary key (_id) as the source. If it does, delete those documents from the destination before starting migration.

• If a DTS task fails, DTS technical support will attempt to restore it within 8 hours. Task parameters (but not database parameters) may be modified during restoration.

Special cases (self-managed source)

• If the source performs a primary-secondary switch during migration, the task fails.

• If the source has no update operations for a long period, the reported latency may be inaccurate. Run an update on the source to refresh the latency, or create a heartbeat that writes data every second.

Step 1: Go to the Data Migration page

Use one of the following methods:

DTS console

1. Log on to the DTS console.

2. In the left-side navigation pane, click Data Migration.

3. In the upper-left corner, select the region where the data migration instance resides.

DMS console

The exact navigation path depends on the mode and layout of your DMS console. For more information, see Simple mode and Customize the layout and style of the DMS console.

1. Log on to the DMS console.

2. In the top navigation bar, choose Data + AI > DTS (DTS) > Data Migration.

3. From the drop-down list next to Data Migration Tasks, select the region where the instance resides.

Step 2: Create a task

1. Click Create Task.

2. (Optional) If New Configuration Page appears in the upper-right corner of the page, click it to switch to the new version. Skip this step if Back to Previous Version is displayed instead.

Step 3: Configure the source and destination databases

Configure the following parameters:

Source database

Destination database

Step 4: Test connectivity

Click Test Connectivity and Proceed.

DTS server CIDR blocks must be added to the security settings of both the source and destination databases. DTS can add them automatically for Alibaba Cloud instances. For self-managed databases, see Add the CIDR blocks of DTS servers. If the source or destination database uses an access method other than Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Step 5: Configure objects to migrate

On the Configure Objects page, set the following parameters:

Step 6: Configure advanced settings

Click Next: Advanced Settings and configure the following:

Step 7: Configure data verification (optional)

Click Next Step: Data Verification to set up a data verification task. For details, see Configure a data verification task.

Step 8: Set default shard key values

Click Next: Configure Database and Table Fields. For each target collection that has shard keys (where Number of Shard Keys is not 0), assign a default value:

1. Click Set Default Value in the row of the target collection.

2. Select a Shard key default value type. Supported types: string and int.

3. Enter the Default Value for the shard key.

Step 9: Run a precheck

Click Next: Save Task Settings and Precheck.

DTS performs a precheck before starting the task. The task can only start after the precheck passes.

If an item fails, click View Details to review the cause, fix the issue, and run the precheck again.

If an item triggers an alert and can be ignored, click Confirm Alert Details, then click Ignore > OK > Precheck Again. Ignoring alerts may lead to data inconsistency.

To preview the API parameters for this configuration, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

Step 10: Purchase and start the instance

1. Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the Purchase Instance page, configure the instance class:

   Parameter	Description
   Resource Group	The resource group for the instance. Default: default resource group. See What is Resource Management?
   Instance Class	Determines migration speed. See Instance classes of data migration instances.

3. Read and select the Data Transmission Service (Pay-as-you-go) Service Terms check box.

4. Click Buy and Start, then click OK in the confirmation dialog box.

Monitor task progress on the Data Migration page.