Data Transmission Service:Migrate a self-managed MongoDB sharded cluster with DTS

Prerequisites

• Verify that your source and destination MongoDB versions are supported. Check supported version combinations in Migration solutions.

• Ensure each shard in the destination sharded cluster instance has sufficient storage space.

  Note

  Example: If the largest shard in your source database uses 500 GB, each shard in the destination instance must have more than 500 GB of storage.

How it works

DTS migrates a sharded cluster one shard at a time. Create a separate migration task for each shard.

Usage notes

• Full data migration increases load on both source and destination databases. Run the migration during off-peak hours to avoid service interruptions.

• When migrating across versions or storage engines, verify compatibility in Versions and storage engines first.

• DTS writes data concurrently, so the destination database may use 5% to 10% more storage than the source.

• Ensure the destination has no documents with the same primary key (_id by default) as the source. If conflicts exist and deletion does not affect your business, delete the conflicting documents from the destination before migration.

• The admin and local databases cannot be used as the source or destination.

• The source MongoDB sharded cluster can have a maximum of 10 mongos nodes.

Billing

Migration types

• Full data migration: Copies all existing data from selected source objects to the destination.

  Note

  Includes databases, collections, and indexes.

• Incremental data migration: After full migration completes, DTS continuously synchronizes data changes from the source to the destination.

  Note

  • DTS synchronizes create and delete operations for databases, collections, and indexes.

  • DTS synchronizes create, delete, and update operations for documents.

Database account permissions

To create database accounts and grant permissions:

• Self-managed MongoDB: db.createUser() (MongoDB documentation).

• ApsaraDB for MongoDB: Manage users of a MongoDB database by using DMS.

Before you begin

1. Required: Disable the balancer during migration to prevent data inconsistencies. Manage the balancer of a MongoDB instance.

   Warning

   If the balancer is not disabled, chunk migrations can affect the consistency of the data DTS reads.

2. Remove orphaned documents created by failed chunk migrations from the source database.

   Note

   Orphaned documents may degrade migration performance and cause _id conflicts that lead to errors.

   1. Download the cleanupOrphaned.js script.

      wget "https://docs-aliyun.cn-hangzhou.oss.aliyun-inc.com/assets/attach/120562/cn_zh/1564451237979/cleanupOrphaned.js"

   2. In the cleanupOrphaned.js script, replace test with the name of the database to clean up.

      Note

      If you have multiple databases, repeat this step and step c.

      function cleanupOrphaned(coll) {
        var nextKey = { };
        var result;
        while ( nextKey != null ) {
          result = db.adminCommand( { cleanupOrphaned: coll, startingFromKey: nextKey } );
          if (result.ok != 1)
            print("Unable to complete at this time: failure or timeout.")
          printjson(result);
          nextKey = result.stoppedAtKey;
        }
      }
      var dbName = 'test'
      db = db.getSiblingDB(dbName)
      db.getCollectionNames().forEach(function(collName) {
              cleanupOrphaned(dbName + "." + collName);
      });

   3. Run this command on each shard to remove orphaned documents from all collections in the database.

      Note

      Run on every shard.

      mongo --host <Shardhost> --port <Primaryport>  --authenticationDatabase <database> -u <username> -p <password> cleanupOrphaned.js

      Note

      • <Shardhost>: The IP address of the shard.

      • <Primaryport>: The service port of the primary node in the shard.

      • <database>: The authentication database for the account.

      • <username>: The database account.

      • <password>: The account password.

      Example with three shards:

      mongo --host 172.16.1.10 --port 27018  --authenticationDatabase admin -u dtstest -p 'Test123456' cleanupOrphaned.js

      mongo --host 172.16.1.11 --port 27021 --authenticationDatabase admin -u dtstest -p 'Test123456' cleanupOrphaned.js

      mongo --host 172.16.1.12 --port 27024  --authenticationDatabase admin -u dtstest -p 'Test123456' cleanupOrphaned.js

3. Create databases and collections that need sharding in the destination instance, and configure data sharding. Configure data sharding to maximize shard performance.

   Note

   Configuring sharding before migration prevents all data from landing on a single shard and exceeding its storage capacity.

Procedure

1. Log on to the DTS console.

2. In the left-side navigation pane, click Data Migration.

3. On the Migration Tasks page, select the region of the destination ApsaraDB for MongoDB instance.

4. In the upper-right corner, click Create Data Migration Task.

5. Configure the source and destination databases.

   Category	Setting	Description
   N/A	Task name	DTS auto-generates a task name. Use a descriptive name for easy identification. Uniqueness is not required.
   Source database	Instance type	Select the instance type based on the source database deployment. This example uses user-created database with public IP address. Note If your self-managed database is a different instance type, you must perform additional preparations. For more information, see Preparations.
   	Instance region	When the instance type is set to user-created database with public IP address, you do not need to configure the Instance Region. Note If your self-managed database uses a whitelist, click Get DTS IP segment next to Instance Region to get DTS server IP addresses and add them to the whitelist.
   	Database type	Select MongoDB.
   	Hostname or IP address	Enter the IP address or domain name of a single shard. This example uses a public IP address. Note DTS migrates one shard at a time. Enter the first shard's information here. Create additional migration tasks for the remaining shards.
   	Port	Enter the shard's service port. Note Each shard's service port must be publicly accessible in this example.
   	Database name	Enter the authentication database name for the account.
   	Database account	Enter the source database account. Required permissions: Database account permissions.
   	Database password	Enter the password for the database account. Note After you enter the source database information, you can click Test Connectivity next to Database Password to verify that the entered information is correct. If the source database information is correct, a Passed message is displayed. If a Failed message is displayed, click Diagnose next to the Failed message, and then adjust the source database information based on the prompts.
   	Connection method	Select Non-encrypted. Note You can select SSL-encrypted only when migrating a MongoDB Atlas database.
   Destination database	Instance type	Select MongoDB instance.
   	Instance region	Select the region of the destination ApsaraDB for MongoDB instance.
   	MongoDB instance ID	Select the ID of the destination sharded cluster instance.
   	Database name	Enter the authentication database name for the account.
   	Database account	Enter the ApsaraDB for MongoDB account. Required permissions: Database account permissions.
   	Database password	Enter the password for the database account. Note After you fill in the destination database information, you can click Test Connectivity next to Database Password to verify that the information is correct. If the information is correct, a Passed message is displayed. If a Failed message is displayed, click Diagnose next to Failed and adjust the destination database information based on the prompt.

6. Click Set whitelist and next.

   If the source or destination database is an Alibaba Cloud database instance, such as an RDS MySQL or ApsaraDB for MongoDB instance, DTS automatically adds the IP addresses of DTS servers in the corresponding region to the instance's whitelist. If the source or destination is a self-managed database on an ECS instance, DTS automatically adds its IP addresses to the ECS security group rules. Ensure the self-managed database also allows access from the ECS instance. If the database is a cluster deployed across multiple ECS instances, you must manually add the IP addresses of DTS servers to the security group rules of each ECS instance. If the source or destination database is a self-managed database in a data center or from another cloud provider, you must manually add the IP addresses of DTS servers to allow access. For a list of DTS server IP addresses, see IP address segments of DTS servers.

   Warning

   Adding public IP addresses of DTS servers, whether automatically or manually, may introduce security risks. By using this product, you acknowledge and accept these potential risks. You are responsible for implementing basic security measures, such as using strong passwords, restricting open ports, using authentication for internal APIs, regularly reviewing and limiting unnecessary network segments, or connecting by using private connections such as Express Connect, VPN Gateway, or Smart Access Gateway.

7. Select the migration type and objects to migrate.

   Parameter	Description
   Migration type	For an offline migration, select only Full Data Migration. For a zero-downtime migration, select both Full Data Migration and Incremental Data Migration. Note If you do not select Incremental Data Migration, do not write new data to the self-managed MongoDB database during the migration. This ensures data consistency between the source and destination.
   Migration objects	In the Available list, click the objects you want to migrate, and then click the icon to move them to the Selected Objects list. Note The admin, local, and config databases cannot be migrated. You can select databases, collections, or functions as migration objects. By default, object names remain the same after migration. To rename an object in the destination database, use the object name mapping feature. For more information, see Object name mapping.
   Specify whether to rename objects	Specifies whether to rename objects in the destination database by using the object name mapping feature. For more information, see Object name mapping.
   Retry duration for unstable source/destination database connections	The default retry duration is 12 hours. You can set a custom duration. If DTS reconnects to the databases within this time, the migration task automatically resumes. Otherwise, the task fails. Note You are charged for the DTS task while it is retrying the connection. We recommend setting the retry duration based on your business needs. To avoid unnecessary charges, release the DTS instance if it is no longer needed.

8. After you complete these settings, click Pre-check and Start in the lower-right corner of the page.

   Note

   • DTS performs a precheck before the task starts. The task can start only after it passes the precheck.

   • If the precheck fails, click the icon next to the failed item to view details.

     • Fix the issues based on the details, and then run the precheck again.

     • For warnings that do not require a fix, you can select Ignore or Ignore and Rerun Precheck to run the precheck again.

9. After the precheck succeeds, click Next.

10. On the Confirm settings page, select an Instance Class and select the Data Transmission Service (Pay-as-you-go) agreement checkbox.

11. Click Buy and Start to start the migration task.

12. Repeat Steps 1 to 11 to create migration tasks for the remaining shards.

13. Complete the migration tasks.

    • Full data migration

      Do not stop the task manually, as this can cause data loss. It stops automatically when complete.

    • Incremental data migration

      The task does not stop automatically. Stop it manually.

      Note

      Stop during off-peak hours or when ready to switch to the ApsaraDB for MongoDB instance.

      1. Wait for all shard migration tasks to show Incremental Data Migration and Undelayed. Then stop writes to the source database for a few minutes. The Incremental Data Migration status may briefly show latency.

      2. When the Incremental Data Migration status for all shard migration tasks returns to Undelayed, stop the tasks manually. Select all migration tasks, confirm that they all show Undelayed, and then click the Stop button at the bottom of the page to complete the migration.

14. Switch your applications to the ApsaraDB for MongoDB instance.