All Products
Search

This Product

    Data Transmission Service:Migrate unsharded MongoDB to a sharded cluster

    Document Center

    Data Transmission Service:Migrate unsharded MongoDB to a sharded cluster

    Last Updated:Jul 07, 2026

    DTS can migrate data to a MongoDB sharded cluster instance and specify a default sharding key value when it is missing from the source. This example uses a source ApsaraDB for MongoDB replica set instance and a destination ApsaraDB for MongoDB sharded cluster instance.

    Prerequisites

    • Created a destination ApsaraDB for MongoDB sharded cluster instance. Create a sharded cluster instance.

      Note

      • If the source ApsaraDB for MongoDB instance uses a sharded cluster architecture, ensure that each shard has an endpoint and that all shards use the same account and password. Apply for a shard endpoint.

      • Supported database versions: Migration scenarios.

    • The storage capacity of the destination ApsaraDB for MongoDB instance must be at least 10% larger than the storage used by the source ApsaraDB for MongoDB instance.

    • Create the databases and collections that require sharding in the destination ApsaraDB for MongoDB instance, configure sharding, enable the balancer, and pre-split the collections. Configure sharding to maximize shard performance. How to handle uneven data distribution in a MongoDB sharded cluster.

      Note

      Sharding prevents data from accumulating on a single shard and maximizes cluster performance. Enabling the balancer and pre-splitting helps prevent data skew.

    Limitations

    Type

    Description

    Source database limitations

    • The server that hosts the source database must have sufficient outbound bandwidth. Otherwise, migration speed will be affected.

    • The collections to be migrated must have a primary key or a unique constraint. Otherwise, data duplication may occur in the destination database.

    • Field names in the data must not contain the "." (dot) character; otherwise, data inconsistencies may occur.

    • If you migrate collections and need to edit their mappings, for example, by mapping collection names, a single migration task supports a maximum of 1,000 collections. Exceeding this limit causes the task submission to fail. In this case, we recommend splitting the collections into multiple tasks or migrating the entire database.

    • A single document in the source database cannot exceed 16 MB. Otherwise, the migration task will fail.

    • If the source MongoDB is a sharded cluster instance, the number of mongos nodes cannot exceed 10.

    • If the source instance is a self-managed MongoDB with a sharded cluster architecture:

      • The Access Method supports only Public IP Address, Express Connect, VPN Gateway, or Smart Access Gateway, and Cloud Enterprise Network (CEN).

      • If the MongoDB version is 8.0 or later and the Migration Method is Oplog, ensure the migration task's shard account has the directShardOperations permission. You can grant this permission by running the following command: db.adminCommand({ grantRolesToUser: "username", roles: [{ role: "directShardOperations", db: "admin"}]}).

        Note

        Replace username in the command with the migration task's shard account.

      • If the Migration Method is Oplog and the task includes full data migration, ensure that the source MongoDB sharded cluster's mongos account has the permission to run the db.runCommand({"balancerStatus":1}) command. DTS uses this command during the precheck phase to verify that the source balancer is disabled.

    • If the source database is an Azure Cosmos DB for MongoDB or an Amazon DocumentDB elastic cluster, only full data migration is supported.

    • To perform incremental data migration:

      The source database must have the oplog enabled with at least seven days of retention. Alternatively, change streams must be enabled, and DTS must be able to subscribe to data changes from the source database within the last seven days by using change streams. If these requirements are not met, the migration task may fail because it cannot obtain data changes from the source. In extreme cases, this can lead to data inconsistency or loss. Issues arising from this are not covered by the DTS Service Level Agreement (SLA).

      Important

      • We recommend using the oplog to obtain data changes from the source database.

      • Only MongoDB 4.0 and later versions support obtaining data changes through change streams.

      • If the source database is an Amazon DocumentDB (non-elastic) cluster, you must manually enable change streams. When you configure the task, set the Migration Method to ChangeStream and the Architecture to Sharded Cluster.

    • Operational limitations on the source database:

      • During the schema migration and full data migration phases, do not perform schema changes on databases or collections, including updating data in arrays. Such changes can cause the migration to fail or lead to data inconsistency between the source and destination databases.

      • If the source MongoDB is a sharded cluster instance, do not run commands that change the data distribution on the source database, such as shardCollection, reshardCollection, unshardCollection, moveCollection, and movePrimary, while the migration is in progress. Otherwise, data inconsistency may occur.

      • If you perform only full data migration, do not write new data to the source instance during migration. Otherwise, data may be inconsistent between the source and destination databases. To ensure real-time data consistency, select schema migration, full data migration, and incremental migration.

      • During migration, do not update fields on the source database that map to the destination shard key. Otherwise, the DTS task may fail.

    • If a collection to be migrated contains a Time-to-Live (TTL) index, data inconsistency may occur or instance latency may increase.

    • Ensure that there are no orphaned documents in the MongoDB sharded cluster instance. Otherwise, data inconsistency or task failure may occur. For more information, see Orphaned documents and How do I clean up orphaned documents from a MongoDB (sharded cluster architecture)?.

    • If the source database is a MongoDB with a sharded cluster architecture, data rebalancing by the balancer may cause instance latency to increase.

    Other limitations

    • For collections that are added to the source database during migration, you cannot set a default value for the shard key.

    • DTS does not support connecting to a MongoDB database by using an SRV record.

    • If the source database is not a sharded cluster and the destination is an Alibaba Cloud MongoDB (sharded cluster architecture), the task proceeds to the Configure Database and Table Fields stage.

    • If the target MongoDB (sharded cluster architecture) is a version earlier than 4.4, the default ShardKey value that you set in the Configure Database and Table Fields step takes effect. DTS then populates the source data with this default value before writing it to the destination. However, if the target MongoDB is version 4.4 or later, the default value that you set in the Configure Database and Table Fields step does not take effect, and DTS writes the source data to the destination as is.

    • For best compatibility, we recommend keeping the source and destination MongoDB versions the same, or migrating from an older version to a newer one. If you migrate from a later version to an earlier version, compatibility issues may occur.

    • DTS does not support migrating data from the admin, config, or local databases.

    • Transaction information is not preserved. Transactions from the source database are converted into individual statements in the destination database.

    • When DTS writes data to a destination collection, if a primary key or unique key conflict occurs, DTS skips the conflicting data write statement and retains the existing data in the destination collection.

    • If the source database is a MongoDB version earlier than 3.6 and the destination database is MongoDB 3.6 or later, field order in the migrated data may differ from the source. The field-value pairs remain correct. This is caused by differences in the database engine's execution plan. If your application logic involves text matching on nested structures, evaluate the potential impact of this field order change.

    • Before you start migration, evaluate the performance of the source and destination databases. We recommend running the migration during off-peak hours. During full data migration, DTS consumes read and write resources from both databases, which may increase their load.

    • During full data migration, DTS performs concurrent INSERT operations. This can cause fragmentation in the destination collections. As a result, the destination collections will use more storage space than the source collections.

    • DTS attempts to resume a failed migration task within seven days. Before you switch your business to the destination instance, you must end or release the task, or revoke the write permissions of the account that DTS uses to access the destination instance. This prevents an automatic task resumption from overwriting data in the destination instance.

    • Because DTS writes data concurrently, the destination uses 5% to 10% more storage space than the source.

    • If the destination collection has a unique index or its capped attribute is set to true, the collection does not support concurrent replay (only single-threaded writing is supported) during incremental data migration. This may increase task latency.

    • The count for the destination MongoDB is queried using the db.$table_name.aggregate([{ $count:"myCount"}]) syntax.

    • To prevent data loss, ensure the destination database does not contain documents with the same primary key (by default, _id) as the source database. If such documents exist, you must clear them from the destination database before migration.

    • If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters.

      Note

      Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters.

    • If you switch traffic to the destination MongoDB database, ensure your business behavior meets MongoDB's requirements for sharded collections.

    • You cannot migrate capped collections when the source database is MongoDB 5.0 or later and the destination database is an earlier version. This can cause the task to fail or result in data inconsistency. This is because the behavior of capped collections changed in MongoDB 5.0, which allows explicit deletions and document size increases on update. Earlier database kernels are not compatible with these features.

    • Time series collections, introduced in MongoDB 5.0, are not supported for migration.

    Special cases

    If the source database is a self-managed MongoDB database:

    • If a primary/secondary switchover occurs on the source database during migration, the task fails.

    • DTS calculates latency by comparing the timestamp of the latest record migrated to the destination database with the current timestamp. If the source database is inactive for a long time, the reported latency may be inaccurate. If the displayed latency is too high, perform a simple update on the source database to get a correct reading.

    Note

    If you migrate an entire database, you can also create a heartbeat by updating or writing data every second.

    Billing

    Migration type

    Instance configuration fee

    Internet traffic fee

    Schema migration and full data migration

    Free of charge.

    When the Access Method parameter of the destination database is set to Public IP Address, you are charged for Internet traffic. For more information, see Billing overview.

    Incremental data migration

    Charged. For more information, see Billing overview.

    Migration types

    Type

    Description

    schema migration

    Migrate the schema of the migration objects from the source ApsaraDB for MongoDB to the destination ApsaraDB for MongoDB.

    Note

    Schema migration is supported for databases, collections, and indexes.

    full data migration

    Migrate all existing data of the migration objects from the source ApsaraDB for MongoDB to the destination ApsaraDB for MongoDB.

    Note

    Full data migration is supported for data in databases and collections.

    incremental data migration

    In addition to the full migration, you can migrate incremental updates from the source ApsaraDB for MongoDB to the destination ApsaraDB for MongoDB.

    Oplog

    Incremental migration does not support databases that are created after the task starts. The following incremental updates are supported:

    • CREATE COLLECTION and CREATE INDEX

    • DROP DATABASE, DROP COLLECTION, and DROP INDEX

      Note

      RENAME COLLECTION operations that include the dropTarget option set to true are not supported.

    • RENAME COLLECTION

    • Inserting, updating, and deleting documents in a collection.

      Note

      For incremental document updates, only changes made by using the $set command are replicated.

    Change stream

    The following incremental updates are supported:

    • DROP DATABASE and DROP COLLECTION

    • RENAME COLLECTION

      Note

      RENAME COLLECTION operations that include the dropTarget option set to true are not supported.

    • Inserting, updating, and deleting documents in a collection.

      Note

      For incremental document updates, only changes made by using the $set command are replicated.

    Database account permissions

    Database

    Schema migration

    Full data migration

    Incremental data migration

    source ApsaraDB for MongoDB instance

    Read permissions on the databases to be migrated and the config database.

    Read permissions on the databases to be migrated, and the admin and local databases.

    destination ApsaraDB for MongoDB instance

    The dbAdminAnyDatabase permission, readWrite permissions on the destination databases, and read permissions on the local and config databases.

    Grant the required permissions on the source and destination ApsaraDB for MongoDB instances. Manage the permissions of MongoDB database users.

    Procedure

    1. Navigate to the migration task list page for the destination region using one of the following methods.

      From the DTS console

      1. Log on to the Data Transmission Service (DTS) console.

      2. In the navigation pane on the left, click Data Migration.

      3. In the upper-left corner of the page, select the region where the migration instance is located.

      From the DMS console

      Note

      The actual operations may vary based on the mode and layout of the DMS console. For more information, see Simple mode console and Customize the layout and style of the DMS console.

      1. Log on to the Data Management (DMS) console.

      2. In the top menu bar, choose Data + AI > Data Transmission (DTS) > Data Migration.

      3. To the right of Data Migration Tasks, select the region where the migration instance is located.

    2. Click Create Task to navigate to the task configuration page.

    3. Configure the source and destination databases.

      Category

      Parameter

      Description

      N/A

      Task Name

      DTS automatically generates a task name. We recommend that you specify a descriptive name for easy identification. The name does not need to be unique.

      Source Database

      Select Existing Connection

      • To use a database instance that has been added to the system (created or saved), select the desired database instance from the drop-down list. The database information below will be automatically configured.

        Note

        In the DMS console, this parameter is named Select a DMS database instance..

      • If you have not registered the database instance with the system, or do not need to use a registered instance, manually configure the database information below.

      Database Type

      Select MongoDB.

      Access Method

      Select Alibaba Cloud Instance.

      Instance Region

      Select the region of the source ApsaraDB for MongoDB instance.

      Replicate Data Across Alibaba Cloud Accounts

      In this example, a database instance under the current Alibaba Cloud account is used. Select No.

      Architecture

      In this example, select Replica Set.

      • Replica Set: High availability and read/write splitting with multiple node types. Replica set instances.

      • Sharded Cluster: Consists of Mongos, Shard, and ConfigServer components with customizable node counts and configurations. Sharded cluster instances.

        Note

        If you set Architecture to Sharded Cluster, you must also specify Shard account and Shard password.

      Migration Method

      Select a method for incremental data migration based on your requirements.

      • Oplog (Recommended):

        This option is available if an oplog is enabled for the source database.

        Note

        An oplog is enabled by default for both self-managed MongoDB databases and ApsaraDB for MongoDB instances. This method offers lower latency for incremental data migration due to faster log pulling. Therefore, we recommend selecting Oplog.

      • ChangeStream: This option is available if Change Streams are enabled for the source database.

        Note

        • If the source database is an Amazon DocumentDB instance (non-elastic cluster), you can only select ChangeStream.

        • If you set Architecture to Sharded Cluster for the source database, you do not need to enter a Shard account or Shard password.

      Instance ID

      Select the ID of the source ApsaraDB for MongoDB instance.

      Authentication Database

      Authentication database of the source ApsaraDB for MongoDB instance. Default: admin.

      Database Account

      Enter the database account of the source ApsaraDB for MongoDB instance. Permissions required for database accounts.

      Database Password

      Enter the password for the specified database account.

      Encryption

      DTS supports three connection methods: Non-encrypted, SSL-encrypted, and Mongo Atlas SSL. The options for Encryption vary based on the selected Access Method and Architecture. The options displayed in the console prevail.

      Note

      • A MongoDB database where the Architecture is Sharded Cluster and the Migration Method is Oplog does not support SSL-encrypted.

      • If the source is a self-managed MongoDB database (Access Method is not Alibaba Cloud Instance) with a Replica Set architecture, and you select SSL-encrypted, DTS also allows you to upload a CA certificate to verify the connection.

      Destination Database

      Select Existing Connection

      • To use a database instance that has been added to the system (created or saved), select the desired database instance from the drop-down list. The database information below will be automatically configured.

        Note

        In the DMS console, this parameter is named Select a DMS database instance..

      • If you have not registered the database instance with the system, or do not need to use a registered instance, manually configure the database information below.

      Database Type

      Select MongoDB.

      Access Method

      Select Alibaba Cloud Instance.

      Instance Region

      Select the region of the destination ApsaraDB for MongoDB instance.

      Replicate Data Across Alibaba Cloud Accounts

      In this example, a database instance under the current Alibaba Cloud account is used. Select No.

      Architecture

      In this example, select Sharded Cluster.

      Instance ID

      Select the ID of the destination ApsaraDB for MongoDB instance.

      Authentication Database

      Authentication database of the destination ApsaraDB for MongoDB instance. Default: admin.

      Database Account

      Enter the database account of the destination ApsaraDB for MongoDB instance. Permissions required for database accounts.

      Database Password

      Enter the password for the specified database account.

      Encryption

      DTS supports three connection methods: Non-encrypted, SSL-encrypted, and Mongo Atlas SSL. The options for Encryption vary based on the selected Access Method and Architecture. The options displayed in the console prevail.

      Note

      • MongoDB databases with an Architecture of Sharded Cluster do not support SSL-encrypted.

      • If the destination is a self-managed MongoDB database (Access Method is not Alibaba Cloud Instance) with a Replica Set, and you select SSL-encrypted, DTS also allows you to upload a CA certificate to verify the connection.

    4. After you complete the configuration, click Test Connectivity and Proceed at the bottom of the page.

      Note

      • Ensure that the IP address segment of the DTS service is automatically or manually added to the security settings of the source and destination databases to allow access from DTS servers. For more information, see Add DTS server IP addresses to a whitelist.

      • If the source or destination database is a self-managed database (the Access Method is not Alibaba Cloud Instance), you must also click Test Connectivity in the CIDR Blocks of DTS Servers dialog box that appears.

    5. Configure the task objects.

      1. On the Configure Objects page, configure the objects that you want to migrate.

        Parameter

        Description

        Migration Types

        • If you only need to perform a full migration, select both Schema Migration and Full Data Migration.

        • To perform a migration with no downtime, select Schema Migration, Full Data Migration, and Incremental Data Migration.

        Note

        • If you do not select Schema Migration, you must ensure that a database and tables to receive the data exist in the destination database. You can also use the object name mapping feature in the Selected Objects box as needed.

        • If you do not select Incremental Data Migration, do not write new data to the source instance during data migration to ensure data consistency.

        Processing Mode of Conflicting Tables

        • Precheck and Report Errors: Checks whether collections with the same names exist in the destination database. If no collections with the same names exist, the precheck is passed. If collections with the same names exist, an error is reported during the precheck, and the data migration task does not start.

          Note

          If a collection in the destination database has the same name but cannot be easily deleted or renamed, you can change the name of the collection in the destination database. For more information, see Object name mapping.

        • Ignore Errors and Proceed: Skips the check for collections with the same names.

          Warning

          Selecting Ignore Errors and Proceed may cause data inconsistency and business risks. For example:

          • If a record in the destination database has the same primary key value as a record in the source database, the record in the destination database is kept. The record from the source database is not migrated to the destination database.

          • Data initialization may fail, only some data may be migrated, or the migration may fail.

        Capitalization of Object Names in Destination Instance

        You can configure the case sensitivity policy for the names of migrated objects, such as databases, tables, and columns, in the destination instance. By default, DTS default policy is selected. You can also choose to keep the case sensitivity consistent with the default policy of the source or destination database. For more information, see Case sensitivity of object names in the destination database.

        Source Objects

        In the Source Objects box, click the objects to migrate, and then click Right arrow to move them to the Selected Objects box.

        Note

        Select databases or collections as migration objects.

        Selected Objects

        • To rename the destination database:

          Right-click the destination database under Selected Objects. In the Edit Schema dialog box, enter the new name in the Schema Name field. Map individual object names.

        • To rename the destination collection:

          Right-click the destination collection under Selected Objects. In the Edit Table dialog box, enter the new name in the Table Name field.

          Important

          Available only when collections are selected as migration objects.

        Note

        • To filter data, right-click the collection in the Selected Objects pane and configure filter conditions. Data filtering is supported for full data migration only. Set filter conditions.

        • Object name mapping may cause dependent object migration to fail.

      2. Click Next: Advanced Settings to configure advanced parameters.

        Parameter

        Description

        Dedicated Cluster for Task Scheduling

        By default, DTS schedules tasks on a shared cluster. You do not need to select one. If you want more stable tasks, you can purchase a dedicated cluster to run DTS migration tasks.

        Retry Time for Failed Connections

        After the migration task starts, if the connection to the source or destination database fails, DTS reports an error and immediately begins to retry the connection. The default retry duration is 720 minutes. You can customize the retry time to a value from 10 to 1440 minutes. We recommend that you set the duration to more than 30 minutes. If DTS reconnects to the source and destination databases within the specified duration, the migration task automatically resumes. Otherwise, the task fails.

        Note

        • For multiple DTS instances that share the same source or destination, the network retry time is determined by the setting of the last created task.

        • Because you are charged for the task during the connection retry period, we recommend that you customize the retry time based on your business needs, or release the DTS instance as soon as possible after the source and destination database instances are released.

        Retry Time for Other Issues

        After the migration task starts, if a non-connectivity issue, such as a DDL or DML execution exception, occurs in the source or destination database, DTS reports an error and immediately begins to retry the operation. The default retry duration is 10 minutes. You can customize the retry time to a value from 1 to 1440 minutes. We recommend that you set the duration to more than 10 minutes. If the related operations succeed within the specified retry duration, the migration task automatically resumes. Otherwise, the task fails.

        Important

        The value of Retry Time for Other Issues must be less than the value of Retry Time for Failed Connections.

        Enable Throttling for Full Data Migration

        During full migration, DTS consumes read and write resources on the source and destination databases, which may increase the database load. If required, you can enable throttling for the full migration task. You can set Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s) to reduce the load on the destination database.

        Note

        • This configuration item is available only if you select Full Data Migration for Migration Types.

        • You can also adjust the full migration speed after the migration instance is running.

        Only one data type for primary key _id in a table of the data to be synchronized

        In the data to be migrated, is the data type of the primary key _id uniform within a single collection?

        Important

        • Select an option based on your requirements. Otherwise, data loss may occur.

        • This parameter is available only if you select Full Data Migration for Migration Types.

        • Yes: The data type is unique. During full data migration, DTS does not scan the data types of primary keys in the source data. For a single collection, DTS migrates only the data corresponding to one primary key data type.

        • No: The data type is not unique. During full data migration, DTS scans the data types of primary keys in the source data and migrates all data.

        Enable Throttling for Incremental Data Migration

        If required, you can also choose to set speed limits for the incremental migration task. You can set RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s) to reduce the load on the destination database.

        Note

        • This configuration item is available only if you select Incremental Data Migration for Migration Types.

        • You can also adjust the incremental migration speed after the migration instance is running.

        Environment Tag

        Optional. Select an environment tag to identify the instance.

        Configure ETL

        Based on your business needs, select whether to configure the ETL feature to process data.

        • Yes: Configures the ETL feature. You must also enter data processing statements in the text box.

        • No: Does not configure the ETL feature.

        Monitoring and Alerting

        Select whether to set alerts and receive alert notifications based on your business needs.

        • No: Does not set an alert.

        • Yes: Configure alerts by setting an alert threshold and an alert notifications. If a migration fails or the latency exceeds the threshold, the system sends an alert notification.

      3. Click Next: Data Validation to configure a data validation task.

        For more information about the data validation feature, see Configure data validation.

      4. Click Next: Configure Database and Table Fields to set the default value for the ShardKey.

        1. In the row of the destination collection, click Set Default Value.

          Note

          If the Number of Shard Keys for a Table Name (collection) is 0, the collection has no shard key and no default value is needed.

        2. Select the Shard key default value type.

          Note

          Shard key default value type supports only string and int.

        3. Set the Default Value for the ShardKey.

          Important

          • The default value takes effect only for destination instances earlier than version 4.4.

          • Set a default value for all ShardKeys of the migration objects. Otherwise, a warning occurs during the Precheck phase and the task may fail.

    6. Save the task and run a precheck.

      • To view the parameters for configuring this instance when you call the API operation, move the pointer over the Next: Save Task Settings and Precheck button and click Preview OpenAPI parameters in the bubble that appears.

      • If you do not need to view or have finished viewing the API parameters, click Next: Save Task Settings and Precheck at the bottom of the page.

      Note

      • Before the migration task starts, DTS performs a precheck. The task starts only after it passes the precheck.

      • If the precheck fails, click View Details next to the failed check item, fix the issue based on the prompt, and then run the precheck again.

      • If a warning is reported during the precheck:

        • For check items that cannot be ignored, click View Details next to the failed item, fix the issue based on the prompt, and then run the precheck again.

        • For check items that can be ignored, you can click Confirm Alert Details, Ignore, OK, and Precheck Again to skip the alert item and run the precheck again. If you choose to ignore a warning, it may cause issues such as data inconsistency and pose risks to your business.

    7. Purchase the instance.

      1. When the Success Rate is 100%, click Next: Purchase Instance.

      2. On the Purchase page, select the link specification for the data migration instance. For more information, see the following table.

        Category

        Parameter

        Description

        New Instance Class

        Resource Group Settings

        Select the resource group to which the instance belongs. The default value is default resource group. For more information, see What is Resource Management?

        Instance Class

        DTS provides migration specifications with different performance levels. The link specification affects the migration speed. You can select a specification based on your business scenario. For more information, see Data migration link specifications.

      3. After the configuration is complete, read and select Data Transmission Service (Pay-as-you-go) Service Terms.

      4. Click Buy and Start. In the OK dialog box that appears, click OK.

        You can view the progress of the migration task on the Data Migration Tasks list page.

        Note

        • If the migration task does not include incremental migration, it stops automatically after the full migration is complete. After the task stops, its Status changes to Completed.

        • If the migration task includes incremental migration, it does not stop automatically. The incremental migration task continues to run. While the incremental migration task is running, the Status of the task is Running.