ApsaraDB for MongoDB:Migrate an ECS-hosted self-managed MongoDB database that uses the sharded cluster architecture to ApsaraDB for MongoDB

How it works

DTS migrates a self-managed MongoDB database by migrating each shard in the database. You must create a data migration task for each shard.

Prerequisites

• The version of the self-managed MongoDB database is 3.0, 3.2, 3.4, 3.6, 4.0, or 4.2.
• Each shard in the destination sharded cluster instance has sufficient storage space.
  Note For example, a self-managed MongoDB database has three shards, and one of these shards occupies the most storage space, which is 500 GB. In this case, the storage space of each shard in the ApsaraDB for MongoDB instance must be greater than 500 GB.

Usage notes

• DTS uses the resources of the source and destination databases during full data migration. This may increase the loads on the database servers. If you migrate a large volume of data or the server specifications cannot meet your requirements, database services may become unavailable. Before you migrate data, evaluate the impact of data migration on the performance of the source and destination databases. We recommend that you migrate data during off-peak hours.
• If the source and destination MongoDB databases have different versions or storage engines, make sure that the versions or storage engines are compatible. For more information, see MongoDB versions and storage engines.
• The data is concurrently written to the destination database. Therefore, the storage space occupied in the destination database is 5% to 10% larger than the size of the data in the source database.
• Make sure that the destination MongoDB database does not have the same primary key as the source database. The default primary key is _id. Otherwise, data loss may occur in the destination database. If the destination database has the same primary key as the source database, clear the related data in the destination database without interrupting the services of DTS. If the same primary key is _id, you can delete the data in the destination database that has the same _id as the source database.
• The admin or local database is not used as the source or destination database.
• The number of mongos nodes in the source MongoDB database that uses the sharded cluster architecture cannot exceed 10.

Billing

Migration types

• All existing data in the source instance is migrated to the destination instance. This is called full data migration.
  Note DTS can migrate the following types of objects: database, collection, and index.
• Incremental data migration: After full data migration is complete, DTS synchronizes incremental data of the source MongoDB database to the destination MongoDB database.
  Note

  • DTS can synchronize the create and delete operations that are performed on databases, collections, and indexes.
  • DTS can synchronize the create, delete, and update operations that are performed on documents.

Permissions required for database accounts

For information about how to create and authorize a database account, see the following topics:

• ApsaraDB for MongoDB instance: Manage user permissions on MongoDB databases
• Self-managed MongoDB database: db.createUser()

Preparations

Procedure

1. Log on to the DTS console.
2. In the left-side navigation pane, click Data Migration.
3. In the Migration Tasks section, select the region where the ApsaraDB for MongoDB instance is deployed.
4. In the upper-right corner, click Create Migration Task.
5. Configure the source and destination databases.

   Section	Parameter	Description
   Task Name	N/A	DTS automatically generates a task name. You do not need to use a unique task name. We recommend that you use an informative name for easy identification.
   Source Database	Instance Type	Select User-Created Database in ECS Instance.
   	Instance Region	The region where the ECS instance is deployed.
   	ECS Instance ID	The ID of the ECS instance. DTS migrates each shard of the source database in turn. In this example, enter the ID of the ECS instance on which the first shard is deployed. For the second migration task, enter the ID of the ECS instance on which the second shard is deployed. You must repeat this procedure until all shards are migrated.
   	Database Type	The type of the database. Select MongoDB.
   	Port Number	The service port of the shard. In this example, enter the service port of the first shard. For the second migration task, enter the service port of the second shard. You must repeat this procedure until all shards are migrated.
   	Database Name	The name of the database corresponding to the account if authentication is enabled.
   	Database Account	The username of the database account used to connect to the source database. For more information about the permissions that are required for the account, see Permissions required for database accounts.
   	Database Password	The password of the database account used to connect to the source database. Note After you specify the information about the source database, you can click Test Connectivity next to Database Password to check whether the information is correct. If the information is correct, the Passed message appears. If the information is incorrect, the Failed message appears and you must click Check next to the Failed message to modify the information.
   Destination Database	Instance Type	Select MongoDB Instance.
   	Instance Region	The region where the destination ApsaraDB for MongoDB instance is deployed.
   	MongoDB Instance ID	The ID of the destination ApsaraDB for MongoDB instance.
   	Database Name	The name of the database corresponding to the account if authentication is enabled.
   	Database Account	The username of the database account used to connect to the destination database. For more information about the permissions that are required for the account, see Permissions required for database accounts.
   	Database Password	The password of the database account used to connect to the destination database. Note After you specify the information about the destination database, you can click Test Connectivity next to Database Password to check whether the information is correct. If the information is correct, the Passed message appears. If the information is incorrect, the Failed message appears. In this case, you must click Check next to the Failed message to modify the information.

6. In the lower-right corner of the page, click Set Whitelist and Next.
   Note

   • You do not need to modify the security settings for ApsaraDB instances (such as ApsaraDB RDS instances and ApsaraDB for MongoDB instances) and ECS-hosted databases. DTS automatically adds the CIDR blocks of DTS servers to the IP whitelists of ApsaraDB instances or the security rules of ECS instances. For more information, see Add the CIDR blocks of DTS servers to the security settings of on-premises databases.
   • If the data source or destination is a self-managed database and the public CIDR blocks of DTS servers are added to the IP whitelist of the self-managed database to allow access from DTS servers, security risks may arise. We recommend that you strengthen authentication with accounts and passwords, restrict allowed ports, or use the internal access methods of Express Connect, VPN Gateway, or Smart Access Gateway.
   • After data migration is complete, you can remove the CIDR blocks of DTS servers from the IP whitelist.
7. Select the migration types and the objects to be migrated.

   Parameter	Description
   Select the migration types	To perform only full data migration, select only Full Data Migration. Note To ensure data consistency, do not write new data into the source database during full data migration. To ensure service continuity during data migration, select Full Data Migration and Incremental Data Migration.
   Select the objects to be migrated	Select one or more objects from the Source Objects section and click the icon to move the objects to the Selected Objects section. Note DTS cannot migrate data from the admin or local database. The config database is an internal database. We recommend that you do not migrate data from the config database. A migration object can be a database, collection, or function. By default, after an object is migrated to the destination instance, the name of the object remains unchanged. You can change the names of the objects in the destination ApsaraDB for MongoDB instance by using the object name mapping feature provided by DTS. For more information, see Object name mapping.

8. In the lower-right corner of the page, click Precheck.
   Note

   • Before you can start the data migration task, DTS performs a precheck. You can start the data migration task only after the task passes the precheck.
   • If the task fails to pass the precheck, click View Details next to each failed item. After you troubleshoot the issues based on the causes, run a precheck again.
   • If an alert is triggered for an item during the precheck:
     • If an alert item cannot be ignored, click View Details next to the failed item and troubleshoot the issues. Then, run a precheck again.
     • If an alert item can be ignored, click Confirm Alert Details. In the View Details dialog box, click Ignore. In the message that appears, click OK. Then, click Precheck Again to run a precheck again. If you ignore the alert item, data inconsistency may occur, and your business may be exposed to potential risks.
9. After the precheck succeeds, click Next.
10. In the Confirm Settings dialog box, specify the Instance Class parameter and select the check box to agree to Data Transmission Service (Pay-As-You-Go) Service Terms.
11. Click Buy and Start to start the data migration task.
12. Repeat step 4 to step 11 to create migration tasks for the remaining shards.
13. Stop the data migration tasks.
    • Full data migration

      We recommend that you do not manually stop the task during full data migration. Otherwise, the data migrated to the destination database may be incomplete. You can wait until the full data migration task automatically stops.

    • Incremental data migration

      An incremental data migration task does not automatically stop. You must manually stop the task.

      Note We recommend that you select an appropriate time to manually stop a data migration task. For example, you can stop the task during off-peak hours or before you switch your workloads to the ApsaraDB for MongoDB instance.
      1. Wait until Incremental Data Migration and The migration task is not delayed are displayed in the progress bar of the data migration task. Then, stop writing data to the source database for a few minutes. The latency of incremental data migration may be displayed in the progress bar.
      2. After the status of incremental data migration changes to The migration task is not delayed again, manually stop the migration tasks for all shards.
14. Switch your workloads to the destination ApsaraDB for MongoDB instance.