edit-icon download-icon

Import data through Data Transmission

Last Updated: Jan 18, 2018

Migration through Data Transmission

ApsaraDB for MongoDB currently provides two migration methods: Data Transmission and MongoDB built-in command line tool. This article describes how to use Data Transmission to import data from the local user-created MongoDB instance to ApsaraDB for MongoDB. Data Transmission supports full migration and incremental migration for smooth MongoDB data migration without stopping local application services. Migration through Data Transmission has following restrictions:

  • Full migration

    Data Transmission migrates all the object data of the source database to the target instance.

  • Incremental migration

    During incremental migration, the incrementally updated data of the local MongoDB instance is synchronized to ApsaraDB for MongoDB, and the local MongoDB and ApsaraDB for MongoDB enter the dynamic synchronization process. Incremental migration achieves smooth migration from local MongoDB to ApsaraDB for MongoDB without interrupting the normal service provision by local MongoDB.

Migration

  • Support for MongoDB 3.0 and 3.2

  • MongoDB full migration

    • Database migration
    • Collection migration
    • Index migration
  • MongoDB incremental migration

    • Synchronization of the Create, Delete, and Update operations on documents
    • Synchronization of the Create and Delete operations on collections
    • Synchronization of the Create and Delete operations on databases
    • Synchronization of the Create and Delete operations on indexes

Migration permission requirements

When Data Transmission is used for migration from MongoDB to ApsaraDB for MongoDB migration, different migration types have different permission requirements for the migration accounts of source and target MongoDB instances, which are listed as follows:

Migration typeFull data migrationIncremental data migration
Local MongoDBread- Read permission of the database to be migrated
- Read permission of admin
- Read permission of the local MongoDB instance
Target MongoDB instancereadWritereadWrite

Preparations before migration

Create a migration account

If you haven’t created a migration account for your local MongoDB or ApsaraDB for MongoDB, create one as follows:

  1. db.createUser({user:"username",pwd:"password",roles:[{role:"rolename1",db:"database_name1"},{role:"rolename2",db:"database_name2"}]})

For permissions required for the migration account, see the preceding section Migration permission requirements.

Parameter description:

  • username: account to be created
  • password: password corresponding to the user name
  • rolename1/rolename2: role name to be assigned to username, for example, read or readWrite
  • database_name1/database_name2: only authorize role1/role2 of database_name1/database_name2 to username.
  • For more information about MongoDB role authorization, see MongoDB Create User Instructions.

Migration steps

To use Data Transmission for data migration, follow these steps:

  1. Log on to the Data Transmission console, and choose Data Migration > Create Migration Task. On the task configuration page, set the task name, source database information, and target database information, as shown in the following figures.

    Source database information

    Target database information

    Information table
    Task name - Data Transmission automatically generates a name for every task. The task name is not required to be unique.
    - You can modify the name as needed. We recommend that you have a name indicating the specific services of the task for easy task identification.
    Source instance information - Instance type: Select Self-built database with a public IP address.
    - Database type: Select MongoDB.
    - Host name or IP address: Set the address for accessing MongoDB. The address must support public access.
    - Port: Listening port of the local MongoDB instance.
    - Database name: Default database name used to connect to the MongoDB instance.
    - Database account: Connection account of the local MongoDB instance.
    - Database password: Password of the connection account of the local MongoDB instance.
    Target instance information - Instance type: Select the MongoDB instance.
    - MongoDB instance ID: Set the instance ID of the target MongoDB instance.
    - Data Transmission supports MongoDB instances of the classic network type. If your MongoDB instances belong to the VPC type, you must change the network type to classic network before migration through Data Transmission.
    - Database name: Default database name used to connect to MongoDB.
    - Database account: Connection account of the MongoDB instance.
    - Database password: Password of the preceding database account.
  2. After configuring the connection information, click Authorize whitelist and enter into next step in the lower-right corner.

    In this step, Data Transmission adds the IP address of the Data Transmission server to the whitelist of the target MongoDB instance to avoid migration failure resulting from failed connection between the Data Transmission server and MongoDB instance because the MongoDB instance is configured with the whitelist.

  3. Select the migration object and migration type.

    Migration object and migration type
    Migration type - MongoDB supports full migration and incremental migration.
    - If you only need full migration, select Full Migration as the migration type.
    - If you need hitless migration, select Full Migration + Incremental Migration as the migration type.
    Migration object - The migration object can be a database or a collection/function.
    - By default, after the object is migrated to the target MongoDB instance, the object name remains the same as that in the local MongoDB instance.
    - If the migrated object has different names on the source and target instances, you must use the object name mapping feature provided by Data Transmission. For more information about the usage, see Database table column mapping.
  4. Perform a pre-check.

    A pre-check is performed before a migration task is formally started. Migration can be started only after the pre-check is successful. For the pre-check content and recovery methods, see Pre-check.

    If the pre-check fails, click the button after the specific check items to view failure details, rectify the faults accordingly, and perform pre-check again.

  5. Start the migration task.

    After a successful pre-check, you can start the migration task. After the task is started, you can view the specific migration status and progress in the task list.

    So far, the configuration of data migration tasks from a local MongoDB instance to ApsaraDB for MongoDB instance is completed.

Pre-check

Data Transmission performs a pre-check before migration. The following table lists the pre-check items for migration from MongoDB to ApsaraDB or MongoDB.

Check itemsCheck contentRemarks
Check source database connectivityCheck the connectivity between the Data Transmission server and local MongoDB instance.- Check whether the entered information is correct. If the entered information contains errors, correct the information and perform pre-check again.
- Check whether the port allows access from other servers.
Check the target database connectionCheck the connectivity between the Data Transmission server and target MongoDB instance.Check whether the entered information is correct. If it is incorrect, correct the information and perform pre-check again.
Check the source library versionCheck whether Data Transmission supports the local MongoDB version.Upgrade MongoDB to version 3.2 and perform pre-check again.
Check source library permissionsCheck whether permissions of the migration account provided by the local MongoDB instance meet the requirements.If this check item fails, see Create a migration account to grant permissions and perform pre-check again.
Check target library permissionsCheck whether permissions of the migration account provided by the target MongoDB instance meet the requirements.If this check item fails, see Create a migration account to grant permissions and perform pre-check again.
Check for incremental topology conflictCheck whether other incremental migration tasks are running on the MongoDB instance.If this check item fails, end or delete other incremental migration tasks and perform pre-check again.
Thank you! We've received your feedback.