Migrate a sharded cluster to a replica set or sharded cluster - Data Transmission Service

Prerequisites

You have created a source ApsaraDB for MongoDB sharded cluster instance and a target ApsaraDB for MongoDB instance (with a replica set or sharded cluster architecture). For more information, see Create a replica set instance and Create a sharded cluster instance.
Note
For information about the supported versions, see Overview of migration solutions.
Obtain an endpoint for each shard node of the source ApsaraDB for MongoDB sharded cluster instance. Ensure that all shards use the same account and password. For more information, see Apply for a shard endpoint.
Before migration, you must disable the balancer on the source ApsaraDB for MongoDB sharded cluster instance. Otherwise, the migration task may fail the precheck. For more information, see Configure data sharding to make full use of shard performance.
The storage space of the target ApsaraDB for MongoDB instance should be at least 10% larger than that used by the source ApsaraDB for MongoDB instance.
If the target ApsaraDB for MongoDB instance uses a sharded cluster architecture, you must create the databases and collections that need to be sharded, configure data sharding, enable the Balancer, and perform pre-sharding in the target ApsaraDB for MongoDB instance based on your business requirements. For more information, see Configure data sharding to maximize shard performance and How to handle uneven data distribution in a MongoDB sharded cluster.
Note
Configuring data sharding prevents data from being migrated to a single shard and ensures optimal cluster performance. Enabling the balancer and performing pre-sharding helps prevent data skew.

Considerations

Type	Description
Source database limitations	Bandwidth requirement: The source database server must have sufficient egress bandwidth. Otherwise, data migration speed is affected. Collections to be migrated must have a primary key or unique constraint, and the constrained fields must contain unique values. Otherwise, duplicate data may be created in the destination database. Field names in the data must not contain the "." (dot) character; otherwise, data inconsistencies may occur. If you migrate data at the collection level and need to edit the collections, such as by mapping collection names, a single migration task can migrate a maximum of 1,000 collections. If you exceed this limit, the task reports an error upon submission. In this case, you should split the collections into multiple batches and configure a separate task for each batch, or configure a task to migrate the entire database. A single document in the source database cannot exceed 16 MB. Otherwise, the migration task will fail. If the source database is an Azure Cosmos DB for MongoDB or an Amazon DocumentDB elastic cluster, only full data migration is supported. To perform incremental data migration: The source database must have the oplog enabled with at least seven days of retention. Alternatively, change streams must be enabled, and DTS must be able to subscribe to data changes from the source database within the last seven days by using change streams. If these requirements are not met, the migration task may fail because it cannot obtain data changes from the source. In extreme cases, this can lead to data inconsistency or loss. Issues arising from this are not covered by the DTS Service Level Agreement (SLA). Important We recommend using the oplog to obtain data changes from the source database. Only MongoDB 4.0 and later versions support obtaining data changes through change streams. If the source database is an Amazon DocumentDB (non-elastic) cluster, you must manually enable change streams. When you configure the task, set the Migration Method to ChangeStream and the Architecture to Sharded Cluster. Operational limitations on the source database: During the schema migration and full data migration phases, do not perform schema changes on databases or collections, including updating data in arrays. Such changes can cause the migration to fail or lead to data inconsistency between the source and destination databases. If you perform only full data migration, do not write new data to the source instance. Otherwise, the source and destination databases become inconsistent. To maintain real-time data consistency, select Schema Migration, Full Data Migration, and Incremental Data Migration. If a collection to be migrated contains a Time-to-Live (TTL) index, data inconsistency may occur or instance latency may increase.
Other limitations	If the destination instance has a sharded cluster architecture: You must clear orphaned documents because they can affect migration performance. If documents with `_id` conflicts are encountered during migration, data inconsistency or task failure can occur. Before starting the task, you must add a shard key to the source data for each sharded collection in the destination. If you cannot add a shard key to the source data, see Migrate data from a MongoDB instance without a shard key to a MongoDB sharded cluster instance. After the task starts, the INSERT command must include the shard key. The UPDATE command cannot change the shard key. If the destination instance has a replica set architecture: When the Access Method is Express Connect, VPN Gateway, or Smart Access Gateway, Public IP Address, or Cloud Enterprise Network (CEN), you must set Domain Name or IP and Port Number to the primary node's address and port, or configure a high-availability connection address. For more information about high-availability connection addresses, see Create a Source or Destination Instance for a High-availability MongoDB Database. When the Access Method is Self-managed Database on ECS, you must set Port Number to the port of the primary node. DTS does not support connecting to a MongoDB database by using an SRV record. Keep the MongoDB versions of the source and destination databases consistent, or migrate from an earlier version to a later version to ensure compatibility. Migrating from a later version to an earlier version may cause compatibility issues. DTS does not support migrating data from the admin, config, or local databases. If the destination collection has a unique index or its `capped` attribute is set to `true`, the collection does not support concurrent replay (only single-threaded writing is supported) during incremental data migration. This may increase task latency. DTS does not preserve transaction information. It converts transactions from the source database into individual statements in the destination database. When DTS writes data to a destination collection, if a primary key or unique key conflict occurs, DTS skips the conflicting data write statement and retains the existing data in the destination collection. If the source database is a MongoDB version earlier than 3.6 and the destination database is MongoDB 3.6 or later, field order in the migrated data may differ from the source. The field-value pairs remain correct. This is caused by differences in the database engine's execution plan. If your application logic involves text matching on nested structures, evaluate the potential impact of this field order change. Before you migrate data, evaluate the performance of both the source and destination databases and perform the migration during off-peak hours. During full data migration, DTS consumes read and write resources, which may increase the database load. During full data migration, DTS performs concurrent INSERT operations, which can cause fragmentation in the destination collections. As a result, the destination collections may occupy more storage space than those in the source instance. Verify that the migration precision DTS uses for FLOAT or DOUBLE columns meets your business requirements. DTS reads values of these types by using `ROUND(COLUMN,PRECISION)`. If you do not explicitly define the precision, DTS migrates FLOAT values with a precision of 38 digits and DOUBLE values with a precision of 308 digits. DTS attempts to resume failed migration tasks within seven days. Before the service cutover to the target instance, you must end or release the task, or use the `revoke` command to revoke the write permissions of the account that DTS uses to access the target instance. This prevents source data from overwriting data on the target instance if the task is automatically resumed. Because DTS writes data concurrently, the destination database uses 5% to 10% more storage space than the source database. Use the `db.$table_name.aggregate([{ $count:"myCount"}])` syntax to query the count in the destination MongoDB. Ensure that the destination MongoDB database does not have documents with the same primary key (the _id field by default) as the source database. Otherwise, data loss may occur. If such documents exist, delete those with conflicting _id values from the destination database before migration, provided that it does not affect your business. If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters. Note Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters. If the destination database is a MongoDB sharded cluster, after you switch your business to this database, you must ensure that your business operations comply with its requirements for sharded collections. You cannot migrate `capped collections` when the source database is MongoDB 5.0 or later and the destination database is an earlier version. This can cause the task to fail or result in data inconsistency. This is because the behavior of `capped collections` changed in MongoDB 5.0, which allows explicit deletions and document size increases on update. Earlier database kernels are not compatible with these features. Time series collections, introduced in MongoDB 5.0, are not supported for migration.
Special cases	If the source database is a self-managed MongoDB database: A primary/secondary switchover on the source database during migration causes the task to fail. DTS calculates latency by comparing the timestamp of the last record migrated to the destination with the current timestamp. If the source database has not been updated for a long time, the reported latency may be inaccurate. If the task shows excessive latency, perform a small update on the source database to refresh the latency value. Note If you migrate an entire database, you can also create a heartbeat table that is updated at regular intervals, such as every second.

Billing

Migration type	Instance configuration fee	Internet traffic fee
Schema migration and full data migration	Free of charge.	When the Access Method parameter of the destination database is set to Public IP Address, you are charged for Internet traffic. For more information, see Billing overview.
Incremental data migration	Charged. For more information, see Billing overview.

Migration types

Type	Description
Schema migration	Migrate the schema of the migration objects from the source ApsaraDB for MongoDB to the destination ApsaraDB for MongoDB. Note Schema migration supports databases, collections, and indexes.
Full data migration	Migrate all existing data of the migration objects from the source ApsaraDB for MongoDB instance to the destination ApsaraDB for MongoDB instance. Note Full data migration supports data in databases and collections.
Incremental data migration	After full data migration, DTS continues to migrate incremental updates from the source ApsaraDB for MongoDB to the target ApsaraDB for MongoDB. Oplog Incremental data migration does not support databases created after the task starts. The following incremental updates are supported: CREATE COLLECTION, INDEX DROP DATABASE, COLLECTION, INDEX Note The `RENAME COLLECTION` operation is not supported if the `dropTarget` option is set to `true`. RENAME COLLECTION Insert, update, and delete documents in a collection. Note When migrating incremental updates for documents, only update operations using the `$set` operator are supported. ChangeStream The following incremental updates are supported: DROP DATABASE, COLLECTION RENAME COLLECTION Note The `RENAME COLLECTION` operation is not supported if the `dropTarget` option is set to `true`. Insert, update, and delete documents in a collection. Note When migrating incremental updates for documents, only update operations using the `$set` operator are supported.

Database account permissions

Database	Schema migration	Full migration	Incremental migration
Source ApsaraDB for MongoDB	Read permission on the databases to be migrated and the config database.		Read permission on the databases to be migrated, the admin database, and the local database.
Target ApsaraDB for MongoDB	The dbAdminAnyDatabase permission, readWrite permission on the target database, and read permission on the local database.

For information about creating and granting permissions to database accounts for the source and destination ApsaraDB for MongoDB instances, see Manage MongoDB database users with DMS.

Clean up orphaned documents

Before migrating data, clean up orphaned documents in the source MongoDB database.

Important

Failure to clean up orphaned documents can degrade migration performance. Conflicting_id values may also cause data migration errors.

ApsaraDB for MongoDB

Note

Running the cleanup script on an instance with a major version earlier than MongoDB 4.2 or a minor version earlier than 4.0.6 results in an error. To view the current version of your instance, see MongoDB Minor Version Release Notes. To upgrade the major or minor version, see Upgrade the major version of an instance and Upgrade the minor version of an instance.

To clean up orphaned documents, use thecleanupOrphaned command. Usage differs by version.

MongoDB 4.4 and later

From a server that can connect to the sharded cluster instance, create a JavaScript script file named cleanupOrphaned.js.

Note

This script cleans up orphaned documents from all collections in multiple databases on multiple shard nodes. To clean up orphaned documents from specific collections, you can modify the script.

// List of shard node IDs
var shardNames = ["shardName1", "shardName2"];
// List of databases to process
var databasesToProcess = ["database1", "database2", "database3"];
shardNames.forEach(function(shardName) {
    // Iterate over the specified list of databases
    databasesToProcess.forEach(function(dbName) {
        var dbInstance = db.getSiblingDB(dbName);
        // Get all collection names for the database instance
        var collectionNames = dbInstance.getCollectionNames();
        // Iterate over each collection
        collectionNames.forEach(function(collectionName) {
            // Full collection name
            var fullCollectionName = dbName + "." + collectionName;
            // Build the cleanupOrphaned command
            var command = {
                runCommandOnShard: shardName,
                command: { cleanupOrphaned: fullCollectionName }
            };
            // Execute the command
            var result = db.adminCommand(command); 
            if (result.ok) {
                print("Cleaned up orphaned documents for collection " + fullCollectionName + " on shard " + shardName);
                printjson(result);
            } else {
                print("Failed to clean up orphaned documents for collection " + fullCollectionName + " on shard " + shardName);
            }
        });
    });
});

Update the values of theshardNames anddatabasesToProcess parameters in the script:

shardNames: An array of IDs for the shard nodes to be cleaned. You can find these IDs in the Shard List area on the Basic Information page of the instance. For example: d-bp15a3796d3a****.
databasesToProcess: An array of names of the databases from which to clean up orphaned documents.

From the directory containing the cleanupOrphaned.js script, run the following command to clean up orphaned documents.

mongo --host <Mongoshost> --port <Primaryport>  --authenticationDatabase <database> -u <username> -p <password> cleanupOrphaned.js > output.txt

The following table describes the parameters.

Parameter	Description
`<Mongoshost>`	The connection address of a mongos node in the sharded cluster instance. For example:`s-bp14423a2a51****.mongodb.rds.aliyuncs.com`.
`<Primaryport>`	The port number of the mongos node. Defaults to 3717.
`<database>`	The database that authenticates the account.
`<username>`	The database account.
`<password>`	The password for the database account.
`output.txt`	Saves the execution output to the output.txt file.

MongoDB 4.2 and earlier

From a server that can connect to the sharded cluster instance, create a JavaScript script file named cleanupOrphaned.js.

Note

This script cleans up orphaned documents from a specific collection in a specific database on multiple shard nodes. To clean up orphaned documents from multiple collections, run the script multiple times with a modifiedfullCollectionName parameter, or modify the script to iterate through the collections.

function cleanupOrphanedOnShard(shardName, fullCollectionName) {
    var nextKey = { };
    var result;
    while ( nextKey != null ) {
        var command = {
            runCommandOnShard: shardName,
            command: { cleanupOrphaned: fullCollectionName, startingFromKey: nextKey }
        };
        result = db.adminCommand(command);
        printjson(result);
        if (result.ok != 1 || !(result.results.hasOwnProperty(shardName)) || result.results[shardName].ok != 1 ) {
            print("Unable to complete at this time: failure or timeout.")
            break
        }
        nextKey = result.results[shardName].stoppedAtKey;
    }
    print("cleanupOrphaned done for coll: " + fullCollectionName + " on shard: " + shardName)
}
var shardNames = ["shardName1", "shardName2", "shardName3"]
var fullCollectionName = "database.collection"
shardNames.forEach(function(shardName) {
    cleanupOrphanedOnShard(shardName, fullCollectionName);
});

Update the values of theshardNames andfullCollectionName parameters in the script:

shardNames: An array of IDs for the shard nodes to be cleaned. You can find these IDs in the Shard List area on the Basic Information page of the instance. For example: d-bp15a3796d3a****.
fullCollectionName: The name of the collection from which to clean up orphaned documents. Use the format<database_name>.<collection_name>.

In the directory where the cleanupOrphaned.js script is located, run the following command to clean up orphaned documents.

mongo --host <Mongoshost> --port <Primaryport>  --authenticationDatabase <database> -u <username> -p <password> cleanupOrphaned.js > output.txt

Parameter	Description
`<Mongoshost>`	The connection address of the mongos node of the sharded cluster instance. Example: `s-bp14423a2a51****.mongodb.rds.aliyuncs.com`.
`<Primaryport>`	The port number of the mongos node of the sharded cluster instance. The default value is 3717.
`<database>`	The name of the authentication database. This is the database to which the database account belongs.
`<username>`	The database account.
`<password>`	The password for the database account.
`output.txt`	The output file for the execution results.

Self-managed MongoDB

From a server that can connect to the self-managed MongoDB database, download the cleanupOrphaned.js script.

wget "https://docs-aliyun.cn-hangzhou.oss.aliyun-inc.com/assets/attach/120562/cn_zh/1564451237979/cleanupOrphaned.js"

Modify the cleanupOrphaned.js script: Replacetest with the name of the database from which you want to clean up orphaned documents.

Important

If you have multiple databases, repeat Step 2 and Step 3 for each database.

function cleanupOrphaned(coll) {
  var nextKey = { };
  var result;
  while ( nextKey != null ) {
    result = db.adminCommand( { cleanupOrphaned: coll, startingFromKey: nextKey } );
    if (result.ok != 1)
       print("Unable to complete at this time: failure or timeout.")
    printjson(result);
    nextKey = result.stoppedAtKey;
  }
}
var dbName = 'test'
db = db.getSiblingDB(dbName)
db.getCollectionNames().forEach(function(collName) {
        cleanupOrphaned(dbName + "." + collName);
});

Run the following command to clean up orphaned documents from all collections in the specified database on a shard node.
Note
You must repeat this step for each shard node.
```
mongo --host <Shardhost> --port <Primaryport>  --authenticationDatabase <database> -u <username> -p <password> cleanupOrphaned.js
```
Note
- <Shardhost>: The IP address of the shard node.
- <Primaryport>: The service port number of the shard's primary node.
- <database>: The database that authenticates the account.
- <username>: The database account.
- <password>: The password for the database account.
Example:
This example assumes a self-managed MongoDB database with three shard nodes. You must run the cleanup command on each node.
```
mongo --host 172.16.1.10 --port 27018  --authenticationDatabase admin -u dtstest -p 'Test123456' cleanupOrphaned.js
```
```
mongo --host 172.16.1.11 --port 27021 --authenticationDatabase admin -u dtstest -p 'Test123456' cleanupOrphaned.js
```
```
mongo --host 172.16.1.12 --port 27024  --authenticationDatabase admin -u dtstest -p 'Test123456' cleanupOrphaned.js
```

Procedure

Navigate to the migration task list page for the destination region using one of the following methods.
From the DTS console
1. Log on to the Data Transmission Service (DTS) console.
2. In the navigation pane on the left, click Data Migration.
3. In the upper-left corner of the page, select the region where the migration instance is located.
From the DMS console

Note
The actual operations may vary based on the mode and layout of the DMS console. For more information, see Simple mode console and Customize the layout and style of the DMS console.
1. Log on to the Data Management (DMS) console.
2. In the top menu bar, choose Data + AI > Data Transmission (DTS) > Data Migration.
3. To the right of Data Migration Tasks, select the region where the migration instance is located.
Click Create Task to navigate to the task configuration page.

Configure the source and destination databases.

Warning

After you select the source and destination instances, we recommend that you carefully read the limits displayed at the top of the page. Otherwise, the task may fail or data inconsistency may occur.

Category	Parameter	Description
N/A	Task Name	DTS automatically generates a task name. We recommend that you specify a descriptive name for easy identification. The name does not need to be unique.
Source Database	Select Existing Connection	To use a database instance that has been added to the system (created or saved), select the desired database instance from the drop-down list. The database information below will be automatically configured. Note In the DMS console, this parameter is named Select a DMS database instance.. If you have not registered the database instance with the system, or do not need to use a registered instance, manually configure the database information below.
	Database Type	Select MongoDB.
	Connection Type	Select Cloud instance.
	Instance Region	Select the region of the source ApsaraDB for MongoDB instance.
	Replicate Data Across Alibaba Cloud Accounts	In this example, a database instance under the current Alibaba Cloud account is used. Select No.
	Architecture	Select Sharded Cluster. Replica Set: Uses multiple node types to achieve high availability and read/write splitting. For details, see Replica set architecture. Sharded Cluster: Provides three components: mongos, shard, and ConfigServer. You can choose the number and configuration of mongos and shard nodes. For details, see Sharded cluster architecture.
	Migration Method	Select a method for incremental data migration. Oplog (Recommended): This option is available if Oplog is enabled on the source database. Note Oplog is enabled by default on both self-managed MongoDB and ApsaraDB for MongoDB instances. This method typically results in lower incremental migration latency because DTS can fetch logs faster. We recommend selecting Oplog. ChangeStream: This option is available if Change Streams is enabled on the source database. Note If the source database is an Amazon DocumentDB instance (non-elastic cluster), you can only select ChangeStream. If you set Architecture to Sharded Cluster for the source database, you do not need to enter a Shard account or Shard password.
	Instance ID	Select the instance ID of the source ApsaraDB for MongoDB instance.
	Authentication Database	Enter the name of the database that the source ApsaraDB for MongoDB instance's account belongs to. The default is `admin`.
	Database Account	Enter the database account for the source ApsaraDB for MongoDB instance. For permission requirements, see Permission requirements for database accounts.
	Database Password	Enter the password for the database account.
	Shard account	If you set Migration Method to Oplog, enter the database account for the shard nodes in the source ApsaraDB for MongoDB sharded cluster instance. Note All shard nodes must use the same account and password. You must also apply for connection strings for the shard nodes in advance. For more information, see Apply for a shard connection string.
	Shard password	If you set Migration Method to Oplog, enter the database password for the shard nodes in the source ApsaraDB for MongoDB sharded cluster instance.
	Encryption	DTS supports three connection methods: Non-encrypted, SSL-encrypted, and Mongo Atlas SSL. The options for Encryption vary based on the selected Access Method and Architecture. The options displayed in the console prevail. Note A MongoDB database where the Architecture is Sharded Cluster and the Migration Method is Oplog does not support SSL-encrypted. If the source is a self-managed MongoDB database (Access Method is not Alibaba Cloud Instance) with a Replica Set architecture, and you select SSL-encrypted, DTS also allows you to upload a CA certificate to verify the connection.
Destination Database	Select Existing Connection	To use a database instance that has been added to the system (created or saved), select the desired database instance from the drop-down list. The database information below will be automatically configured. Note In the DMS console, this parameter is named Select a DMS database instance.. If you have not registered the database instance with the system, or do not need to use a registered instance, manually configure the database information below.
	Database Type	Select MongoDB.
	Connection Type	Select Cloud instance.
	Instance Region	Select the region of the destination ApsaraDB for MongoDB instance.
	Replicate Data Across Alibaba Cloud Accounts	In this example, a database instance under the current Alibaba Cloud account is used. Select No.
	Architecture	Select an architecture based on your business requirements. Valid values: Replica Set: Uses multiple node types to achieve high availability and read/write splitting. For details, see Replica set architecture. Sharded Cluster: Provides three components: mongos, shard, and ConfigServer. You can choose the number and configuration of mongos and shard nodes. For details, see Sharded cluster architecture.
	Instance ID	Select the instance ID of the destination ApsaraDB for MongoDB instance.
	Authentication Database	Enter the name of the database that the destination ApsaraDB for MongoDB instance's account belongs to. The default is `admin`.
	Database Account	Enter the database account for the destination ApsaraDB for MongoDB instance. For permission requirements, see Permission requirements for database accounts.
	Database Password	Enter the password for the database account.
	Encryption	DTS supports three connection methods: Non-encrypted, SSL-encrypted, and Mongo Atlas SSL. The options for Encryption vary based on the selected Access Method and Architecture. The options displayed in the console prevail. Note MongoDB databases with an Architecture of Sharded Cluster do not support SSL-encrypted. If the destination is a self-managed MongoDB database (Access Method is not Alibaba Cloud Instance) with a Replica Set, and you select SSL-encrypted, DTS also allows you to upload a CA certificate to verify the connection.

After completing the configuration, click Test Connectivity and Proceed at the bottom of the page.
Note
- Ensure that you add the CIDR blocks of the DTS servers (either automatically or manually) to the security settings of both the source and destination databases to allow access. For more information, see Add the IP address whitelist of DTS servers.
- If the source or destination is a self-managed database (i.e., the Access Method is not Alibaba Cloud Instance), you must also click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Configure the task objects.

On the Configure Objects page, configure the objects that you want to migrate.

Parameter	Description
Migration Types	If you only need to perform a full migration, select both Schema Migration and Full Data Migration. To perform a migration with no downtime, select Schema Migration, Full Data Migration, and Incremental Data Migration. Note If you do not select Schema Migration, you must ensure that a database and tables to receive the data exist in the destination database. You can also use the object name mapping feature in the Selected Objects box as needed. If you do not select Incremental Data Migration, do not write new data to the source instance during data migration to ensure data consistency. For more information, see Migration types.
Processing Mode of Conflicting Tables	Precheck and Report Errors: Checks whether collections with the same names exist in the destination database. If no collections with the same names exist, the precheck is passed. If collections with the same names exist, an error is reported during the precheck, and the data migration task does not start. Note If a collection in the destination database has the same name but cannot be easily deleted or renamed, you can change the name of the collection in the destination database. For more information, see Object name mapping. Ignore Errors and Proceed: Skips the check for collections with the same names. Warning Selecting Ignore Errors and Proceed may cause data inconsistency and business risks. For example: If a record in the destination database has the same primary key value as a record in the source database, the record in the destination database is kept. The record from the source database is not migrated to the destination database. Data initialization may fail, only some data may be migrated, or the migration may fail.
Capitalization of Object Names in Destination Instance	You can configure the case sensitivity policy for the names of migrated databases and collections in the destination instance. By default, DTS default policy is selected. You can also choose to align with the default policies of the source or destination database. For more information, see Case sensitivity policy for destination object names.
Source Objects	In the Source Objects box, select an object to migrate, and then click to move it to the Selected Objects box. Note Migration objects can be selected at the DATABASE or COLLECTION level.
Selected Objects	To set the name of a migration object in the destination instance, or to specify the object that receives data in the destination instance, right-click the migration object in the Selected Objects box to make changes. For more information, see Object name mapping. To remove a selected migration object, click the object in the Selected Objects box, and then click to move it to the Source Objects box. Note To select incremental migration operations at the database or collection level, right-click the object in the Selected Objects box and make your selections in the dialog box that appears. To filter data by using conditions (supported for full data migration but not for incremental data migration), right-click the collection in the Selected Objects box and configure the settings in the dialog box. For instructions, see Set filter conditions. If you use the object name mapping feature to specify a database or collection to receive data, the migration of other objects that depend on the mapped object may fail.

Click Next: Advanced Settings to configure advanced parameters.

Parameter	Description
Dedicated Cluster for Task Scheduling	By default, DTS schedules tasks on a shared cluster. You do not need to select one. If you want more stable tasks, you can purchase a dedicated cluster to run DTS migration tasks.
Retry Time for Failed Connections	After the migration task starts, if the connection to the source or destination database fails, DTS reports an error and immediately begins to retry the connection. The default retry duration is 720 minutes. You can customize the retry time to a value from 10 to 1440 minutes. We recommend that you set the duration to more than 30 minutes. If DTS reconnects to the source and destination databases within the specified duration, the migration task automatically resumes. Otherwise, the task fails. Note For multiple DTS instances that share the same source or destination, the network retry time is determined by the setting of the last created task. Because you are charged for the task during the connection retry period, we recommend that you customize the retry time based on your business needs, or release the DTS instance as soon as possible after the source and destination database instances are released.
Retry Time for Other Issues	After the migration task starts, if a non-connectivity issue, such as a DDL or DML execution exception, occurs in the source or destination database, DTS reports an error and immediately begins to retry the operation. The default retry duration is 10 minutes. You can customize the retry time to a value from 1 to 1440 minutes. We recommend that you set the duration to more than 10 minutes. If the related operations succeed within the specified retry duration, the migration task automatically resumes. Otherwise, the task fails. Important The value of Retry Time for Other Issues must be less than the value of Retry Time for Failed Connections.
Enable Throttling for Full Data Migration	During full migration, DTS consumes read and write resources on the source and destination databases, which may increase the database load. If required, you can enable throttling for the full migration task. You can set Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s) to reduce the load on the destination database. Note This configuration item is available only if you select Full Data Migration for Migration Types. You can also adjust the full migration speed after the migration instance is running.
Only one data type for primary key _id in a table of the data to be synchronized	Indicates whether the data type of the primary key `_id` is unique within each collection to be migrated. Important Select an option based on your actual data. An incorrect selection may lead to data loss. This parameter is available only if you select Full Data Migration for Migration Types. Yes: The data type is unique. During full data migration, DTS does not scan the primary key data types in the source data. For each collection, DTS migrates only the data corresponding to one primary key data type. No: The data type is not unique. During full data migration, DTS scans the primary key data types in the source data and migrates all corresponding data.
Enable Throttling for Incremental Data Migration	If required, you can also choose to set speed limits for the incremental migration task. You can set RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s) to reduce the load on the destination database. Note This configuration item is available only if you select Incremental Data Migration for Migration Types. You can also adjust the incremental migration speed after the migration instance is running.
Environment Tag	You can select an environment tag to identify the instance based on your needs. This is not required for this example.
Configure ETL	Based on your business needs, select whether to configure the ETL feature to process data. Yes: Configures the ETL feature. You must also enter data processing statements in the text box. No: Does not configure the ETL feature.
Monitoring and Alerting	Select whether to set alerts and receive alert notifications based on your business needs. No: Does not set an alert. Yes: Configure alerts by setting an alert threshold and an alert notifications. If a migration fails or the latency exceeds the threshold, the system sends an alert notification.

Click Next: Data Validation to configure a data validation task.

For more information about the data validation feature, see Configure data validation.

Save the task and run a precheck.
- To view the parameters for configuring this instance when you call the API operation, move the pointer over the Next: Save Task Settings and Precheck button and click Preview OpenAPI parameters in the bubble that appears.
- If you do not need to view or have finished viewing the API parameters, click Next: Save Task Settings and Precheck at the bottom of the page.
Note
- Before the migration task starts, DTS performs a precheck. The task starts only after it passes the precheck.
- If the precheck fails, click View Details next to the failed check item, fix the issue based on the prompt, and then run the precheck again.
- If a warning is reported during the precheck:
  
  For check items that cannot be ignored, click View Details next to the failed item, fix the issue based on the prompt, and then run the precheck again.
  
  For check items that can be ignored, you can click Confirm Alert Details, Ignore, OK, and Precheck Again to skip the alert item and run the precheck again. If you choose to ignore a warning, it may cause issues such as data inconsistency and pose risks to your business.

Purchase the instance.

When the Success Rate is 100%, click Next: Purchase Instance.

On the Purchase page, select the link specification for the data migration instance. For more information, see the following table.

Category	Parameter	Description
New Instance Class	Resource Group Settings	Select the resource group to which the instance belongs. The default value is default resource group. For more information, see What is Resource Management?
New Instance Class	Instance Class	DTS provides migration specifications with different performance levels. The link specification affects the migration speed. You can select a specification based on your business scenario. For more information, see Data migration link specifications.

After the configuration is complete, read and select Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start. In the OK dialog box that appears, click OK.

You can view the progress of the migration task on the Data Migration Tasks list page.
Note
- If the migration task does not include incremental migration, it stops automatically after the full migration is complete. After the task stops, its Status changes to Completed.
- If the migration task includes incremental migration, it does not stop automatically. The incremental migration task continues to run. While the incremental migration task is running, the Status of the task is Running.

FAQ

Why do I encounter task latency and data inconsistency even when there are no application writes?

Cause: This issue occurs because of a conflict between the automatic deletion mechanism of a TTL index on a MongoDB collection and the data synchronization mechanism of DTS. This conflict can lead to task latency and data inconsistency in your synchronization/migration task.

Redundant DELETEs reduce efficiency: When the source TTL index deletes expired data, it writes a DELETE record to the Oplog. DTS replays this DELETE on the destination. If the destination TTL index already deleted the same data, MongoDB returns an unexpected affected-row count, triggering exception handling and slowing the migration.

Data inconsistency from asynchronous TTL deletion: TTL indexes do not delete data in real time. Expired data may still exist on the source while the destination has already deleted it, causing inconsistency.

Example:

The MongoDB Oplog or ChangeStream records only the updated fields for an UPDATE operation, not the full document. If an UPDATE cannot find the target data on the destination, DTS ignores the operation.

Timing	Source instance	Destination instance
1	Service inserts data
2		DTS synchronizes the INSERT operation
3	Data has expired but is not yet deleted by the TTL index
4	Service updates the data (for example, updates the TTL index field to change the expiration time)
5		TTL index deletes the data
6		DTS synchronizes the UPDATE, but the data is not found. The operation is ignored.

As a result, this document is missing from the destination MongoDB instance.

Solution: To resolve this, temporarily modify the expiration time of the TTL index on the target during the synchronization/migration task. This ensures both synchronization efficiency and data consistency. For detailed steps, see Best practices for synchronizing or migrating collections with TTL indexes from a MongoDB source.

Prerequisites

Considerations

Billing

Migration types

Oplog

ChangeStream

Database account permissions

Clean up orphaned documents

ApsaraDB for MongoDB

MongoDB 4.4 and later

MongoDB 4.2 and earlier

Self-managed MongoDB

Procedure

From the DTS console

From the DMS console

FAQ