ApsaraDB for MongoDB:Migrate data from an ECS-hosted self-managed MongoDB database that uses the sharded cluster architecture to ApsaraDB for MongoDB

How it works

DTS migrates a sharded cluster by processing each shard separately. You must create one data migration task for each shard.

Data distribution in the destination ApsaraDB for MongoDB instance is based on the shard key you specify. For more information, see Configure sharding to maximize the performance of shards.

Migration types

Run both migration types together to keep your applications running during migration.

If you run full data migration only, stop writing to the source database until the task completes — otherwise the migrated data will be incomplete.

Prerequisites

Before you begin, make sure that you have:

• A self-managed MongoDB database running MongoDB 3.0 or later

• Sufficient storage space on each destination shard — each destination shard must be larger than the largest shard in the source database (for example, if the largest source shard is 500 GB, each destination shard must exceed 500 GB)

• Database accounts with the permissions described in Permissions required for database accounts

Permissions required for database accounts

To create accounts and grant permissions:

• ApsaraDB for MongoDB instances: Manage user permissions on MongoDB databases

• Self-managed MongoDB databases: db.createUser()

Billing

Usage notes

• DTS uses the resources of the source and destination databases during full data migration. This may increase the loads on the database servers. Migrate data during off-peak hours to reduce the risk of performance degradation.

• If the source and destination databases run different MongoDB versions or use different storage engines, verify compatibility before proceeding. See MongoDB versions and storage engines.

• Due to concurrent writes, the destination database uses 5%–10% more storage space than the source data size.

• The destination ApsaraDB for MongoDB instance must not contain documents with the same _id primary key as the source. If duplicate _id values exist, delete the conflicting documents from the destination before starting migration — otherwise data may be lost.

• Do not use the admin or local database as the source or destination.

• The source sharded cluster must have no more than 10 mongos nodes.

Prepare the source database

Before creating migration tasks, disable the balancer on the source database and delete orphaned documents. Disabling the balancer prevents chunk migrations from interfering with data consistency during the DTS migration. For instructions, see Migrate a self-managed MongoDB database that uses the sharded cluster architecture to ApsaraDB for MongoDB using DTS.

Migrate data from each shard

Create one migration task per shard. Repeat the following steps for each shard until all shards are migrated.

1. Log on to the DTS console.

2. In the left navigation pane, click Data Migration.

3. At the top of the Data Migration Tasks page, select the region where the destination ApsaraDB for MongoDB instance resides.

4. In the upper-right corner, click Create Task.

5. Configure the source and destination databases using the following parameters.

   Source database

   Parameter	Description
   Instance type	Select Self-managed Database on ECS.
   Instance region	Select the region where the ECS instance is deployed.
   ECS instance ID	Enter the ECS instance ID that hosts the current shard. For the first task, enter the first shard's ECS instance ID. For each subsequent task, enter the ECS instance ID for the next shard.
   Database type	Select MongoDB.
   Port number	Enter the service port of the current shard.
   Database name	Enter the name of the authentication database.
   Database account	Enter the account for connecting to the source database. See Permissions required for database accounts.
   Database password	Enter the password for the database account.

   Destination database

   Parameter	Description
   Instance type	Select MongoDB Instance.
   Instance region	Select the region where the destination ApsaraDB for MongoDB instance resides.
   MongoDB instance ID	Select the destination ApsaraDB for MongoDB instance.
   Database name	Enter the name of the authentication database.
   Database account	Enter the account for connecting to the destination database. See Permissions required for database accounts.
   Database password	Enter the password for the database account. After entering the destination database information, click Test Connectivity and Proceed to validate. Passed appears if the connection is valid. If Failed appears, click Check next to the Failed message to fix the configuration.

6. In the lower-right corner, click Set Whitelist and Next.

   DTS automatically adds its CIDR blocks to the IP whitelists of Alibaba Cloud database instances and to the security rules of ECS instances — no manual changes are needed. If the data source or destination is a self-managed database and the public CIDR blocks of DTS servers are added to the IP whitelist of the self-managed database, security risks may occur. We recommend that you strengthen authentication with accounts and passwords, restrict allowed ports, or connect to the database over Express Connect, VPN Gateway, or Smart Access Gateway. After migration is complete, remove the DTS CIDR blocks from your whitelists. For more information, see Add the CIDR blocks of DTS servers to the security settings of on-premises databases.

7. Select the migration types and objects to migrate.

   Parameter	Description
   Migration type	Select Full Data Migration only for a one-time migration. Select both Full Data Migration and Incremental Data Migration to keep the destination in sync while your applications run.
   Source objects	In the Source Objects section, select the databases, collections, or functions to migrate, then click the arrow icon to move them to the Selected Objects list. Note DTS cannot migrate data from the admin or local database. Migrating data from the config database is not recommended, as it is an internal database. To rename objects in the destination, use the Object name mapping feature.

8. In the lower-right corner, click Next: Save Task Settings and Precheck. DTS runs a precheck before starting migration. If any item fails, click View Details next to the failed item, fix the issue, and then click Precheck Again. If an alert appears, click Confirm Alert Details, then Ignore, then OK, and then Precheck Again.

   Important

   Ignoring a precheck alert may cause data inconsistency. Proceed only if you understand the risk.

9. Click Next: Purchase Instance.

10. In the Purchase dialog box, set the Instance Class, then select the checkbox to agree to Data Transmission Service (Pay-As-You-Go) Service Terms.

11. Click Buy and Start to start the migration task.

12. Repeat steps 4–11 for each remaining shard.

Stop the migration tasks

After all shard tasks are running, stop them based on the migration type.

Full data migration

Do not manually stop a full data migration task — wait for it to complete automatically. Stopping it early results in incomplete data in the destination.

Incremental data migration

Incremental migration tasks do not stop automatically. Stop them manually after the lag drops to zero:

1. Watch the progress bar of each shard's migration task. Wait until both Incremental Data Migration and The migration task is not delayed are displayed.

2. Stop writing data to the source database for a few minutes.

3. After the incremental migration status shows The migration task is not delayed again, manually stop the migration tasks for all shards. Stop all shard tasks during off-peak hours or just before you switch workloads to the destination.

   

Switch workloads to the destination

After all migration tasks are stopped, switch your applications to the destination ApsaraDB for MongoDB instance.