This topic describes how to use Data Transmission Service (DTS) to migrate an ECS-hosted self-managed MongoDB database that uses the sharded cluster architecture to an ApsaraDB for MongoDB instance. DTS allows you to migrate historical and incremental data without business interruptions.

How it works

DTS migrates a self-managed MongoDB database by migrating each shard in the database. You must create a data migration task for each shard.

Note The distribution of migrated data in the destination ApsaraDB for MongoDB instance is based on the shard key that you specify. For more information, see Configure sharding to maximize the performance of shards.
How it works

Prerequisites

  • The version of the self-managed MongoDB database is 3.0, 3.2, 3.4, 3.6, 4.0, or 4.2.
  • Each shard in the destination sharded cluster instance has sufficient storage space.
    Note For example, a self-managed MongoDB database has three shards, and one of these shards occupies the most storage space, which is 500 GB. In this case, the storage space of each shard in the ApsaraDB for MongoDB instance must be greater than 500 GB.

Precautions

  • DTS consumes the resources of the source and destination databases during full data migration. This may increase the loads of the database servers. If you migrate a large volume of data or the server specifications cannot meet your requirements, database services may become unavailable. Before you migrate data, evaluate the impact of data migration on the performance of the source and destination databases. We recommend that you migrate data during off-peak hours.
  • If the source and destination MongoDB databases use different versions or storage engines, make sure that your applications can run on both databases. For more information about the versions and storage engines that are supported by ApsaraDB for MongoDB, see MongoDB versions and storage engines.

Billing information

Migration Types Instance configurations Internet traffic
Full data migration Free of charge Free of charge
Incremental data migration Charged, For more information, see Data Transmission Service (DTS) pricing. Free of charge

Migration types

  • All existing data in the source instance is migrated to the destination instance. This is called full data migration.
    Note DTS can migrate the following types of objects: database, collection, and index.
  • Incremental data migration: After full data migration is complete, DTS synchronizes incremental data of the source MongoDB database to the destination MongoDB database.
    Note
    • DTS can synchronize the create and delete operations that are performed on databases, collections, and indexes.
    • DTS can synchronize the create, delete, and update operations that are performed on documents.

Permissions required for database accounts

Database Full data migration Incremental data migration
Self-managed MongoDB database on ECS Read permissions on the source database The read permissions on the source database, the admin database, and the local database
Destination ApsaraDB for MongoDB instance Read and write permissions on the destination database Read and write permissions on the destination database

For information about how to create and authorize a database account, see the following topics:

Preparations

Disable the balancer for the source database and delete orphaned documents. For more information, see Migrate a user-created sharded MongoDB database to ApsaraDB for MongoDB by using DTS.

Procedure

  1. Log on to the DTS console.
  2. In the left-side navigation pane, click Data Migration.
  3. In the Migration Tasks section, select the region where the ApsaraDB for MongoDB instance is deployed.
  4. In the upper-right corner, click Create Migration Task.
  5. Configure the source and destination databases.
    Section Parameter Description
    Task Name N/A
    • DTS automatically generates a task name. You do not need to use a unique task name.
    • We recommend that you use an informative name for easy identification.
    Source Database Instance Type Select User-Created Database in ECS Instance.
    Instance Region The region where the ECS instance is deployed.
    ECS Instance ID

    The ID of the ECS instance. DTS migrates each shard of the source database in turn. In this example, enter the ID of the ECS instance on which the first shard is deployed.

    For the second migration task, enter the ID of the ECS instance on which the second shard is deployed. You must repeat this procedure until all shards are migrated.

    Database Type The type of the database. Select MongoDB.
    Port Number

    The service port of the shard. In this example, enter the service port of the first shard.

    For the second migration task, enter the service port of the second shard. You must repeat this procedure until all shards are migrated.

    Database Name The name of the database corresponding to the account if authentication is enabled.
    Database Account The username of the database account used to connect to the source database. For more information about the permissions that are required for the account, see Permissions required for database accounts.
    Database Password The password of the database account used to connect to the source database.
    Note After you specify the information about the source database, you can click Test Connectivity next to Database Password to check whether the information is correct. If the information is correct, the Passed message appears. If the information is incorrect, the Failed message appears and you must click Check next to the Failed message to modify the information.
    Destination Database Instance Type Select MongoDB Instance.
    Instance Region The region where the destination ApsaraDB for MongoDB instance is deployed.
    MongoDB Instance ID The ID of the destination ApsaraDB for MongoDB instance.
    Database Name The name of the database corresponding to the account if authentication is enabled.
    Database Account The username of the database account used to connect to the destination database. For more information about the permissions that are required for the account, see Permissions required for database accounts.
    Database Password The password of the database account used to connect to the destination database.
    Note After you specify the information about the destination database, you can click Test Connectivity next to Database Password to check whether the information is correct. If the information is correct, the Passed message appears. If the information is incorrect, the Failed message appears. In this case, you must click Check next to the Failed message to modify the information.
  6. In the lower-right corner of the page, click Set Whitelist and Next.
    Note
    • You do not need to modify the security settings for ApsaraDB instances (such as ApsaraDB RDS instances and ApsaraDB for MongoDB instances) and ECS-hosted databases. DTS automatically adds the CIDR blocks of DTS servers to the IP whitelists of ApsaraDB instances or the security rules of ECS instances. For more information, see Add the CIDR blocks of DTS servers to the security settings of on-premises databases.
    • If the data source or destination is a self-managed database and the public CIDR blocks of DTS servers are added to the IP whitelist of the self-managed database to allow access from DTS servers, security risks may arise. We recommend that you strengthen authentication with accounts and passwords, restrict allowed ports, or use the internal access methods of Express Connect, VPN Gateway, or Smart Access Gateway.
    • After data migration is complete, you can remove the CIDR blocks of DTS servers from the IP whitelist.
  7. Select the migration types and the objects to be migrated. MongoDB: Select the migration types and the objects to be migrated
    Parameter Description
    Select the migration types
    • To perform only full data migration, select only Full Data Migration.
      Note To ensure data consistency, do not write new data into the source database during full data migration.
    • To ensure service continuity during data migration, select Full Data Migration and Incremental Data Migration.
    Select the objects to be migrated
    • Select one or more objects from the Source Objects section and click the icon to move the objects to the Selected Objects section.
      Note
      • DTS cannot migrate data from the admin or local database.
      • The config database is an internal database. We recommend that you do not migrate data from the config database.
    • A migration object can be a database, collection, or function.
    • By default, after an object is migrated to the destination instance, the name of the object remains unchanged. You can change the names of the objects in the destination ApsaraDB for MongoDB instance by using the object name mapping feature provided by DTS. For more information, see Object name mapping.
  8. In the lower-right corner of the page, click Precheck.
    Note
    • A precheck is performed before the migration task starts. The migration task starts only after the precheck succeeds.
    • If the precheck fails, click the Tip icon for each failed check item to view their details. Perform a precheck again after the failures are fixed.
  9. After the precheck succeeds, click Next.
  10. Click Buy and Start to start the data migration task.
  11. Repeat step 4 to step 11 to create migration tasks for the remaining shards.
  12. Stop the data migration tasks.
    • Full data migration

      We recommend that you do not manually stop the task during full data migration. Otherwise, the data migrated to the destination database will be incomplete. You can wait until the data migration task automatically stops.

    • Incremental data migration

      A task does not automatically stop during incremental data migration. You must manually stop the task.

      Note We recommend that you select an appropriate time to manually stop a data migration task. For example, you can stop the task during off-peak hours or before you switch your workloads to the ApsaraDB for MongoDB instance.
      1. Wait until Incremental Data Migration and The migration task is not delayed appear in the progress bar of all migration tasks. Then, stop writing data to the source database for a few minutes. The delay time of incremental data migration may be displayed in the progress bar.
      2. After the status of incremental data migration changes to The migration task is not delayed again, manually stop the migration tasks for all shards. Stop the data migration tasks
  13. Switch your workloads to the destination ApsaraDB for MongoDB instance.