Use Data Transmission Service (DTS) to migrate a self-managed MongoDB replica set database to ApsaraDB for MongoDB. DTS supports full and incremental data migration, so you can complete the migration without interrupting your services.
For an alternative approach, see Migrate self-managed databases to Alibaba Cloud using MongoDB tools. For a full list of migration and synchronization options, see Data migration and synchronization.
Prerequisites
Before you begin, make sure that:
The source self-managed MongoDB database and the destination ApsaraDB for MongoDB instance are running supported versions. For details, see Overview of migration solutions.
The destination ApsaraDB for MongoDB instance has at least 10% more storage space than the source database.
The database accounts meet the permission requirements in the Database account permissions section below.
Billing
| Migration type | Task configuration fee | Data transfer cost |
|---|---|---|
| Full data migration | Free | Charged if you transfer data from Alibaba Cloud over the Internet. See Billing overview. |
| Incremental data migration | Charged. See . | — |
Migration types
Full data migration copies all historical data from the source to the destination. The following objects can be migrated: databases, collections, and indexes.
Incremental data migration replicates ongoing changes after the full migration completes. The following operations are synchronized:
Create and delete operations on databases, collections, and indexes
Create, delete, and update operations on documents
To migrate without downtime, select both full and incremental data migration.
Usage notes
Data restrictions:
Do not migrate the
configdatabase unless you have a specific reason to. It is an internal system database.The
adminandlocaldatabases cannot be the source or destination.Make sure the destination database has no documents with the same
_idas documents in the source database. Duplicate primary keys cause data loss. If conflicts exist, delete the conflicting documents from the destination before starting the migration.
Storage:
DTS writes data concurrently, so the destination database may use 5%–10% more storage space than the source.
Performance:
Full data migration consumes read and write resources from both databases and may increase server load. Run the migration during off-peak hours to reduce the impact on production traffic.
Compatibility:
For supported MongoDB versions and storage engines, see Versions and storage engines. Check for compatibility issues before migrating across different versions or storage engines.
Database account permissions
| Data source | Full data migration | Incremental data migration |
|---|---|---|
| Self-managed MongoDB database | read on the database to migrate | read on the database to migrate, the admin database, and the local database |
| ApsaraDB for MongoDB | readWrite on the destination database | readWrite on the destination database |
To create accounts and assign permissions:
ApsaraDB for MongoDB instance: See Manage MongoDB database users in Data Management Service (DMS).
Self-managed MongoDB database: Use
db.createUser()to create the required account. Example:use admin db.createUser({ user: "dts_user", pwd: "yourPassword", roles: [ { role: "read", db: "your_source_database" }, { role: "read", db: "admin" }, { role: "read", db: "local" } ] })For full syntax, see Create User in MongoDB.
Migrate the database
Step 1: Open the Data Migration page
Use one of the following consoles:
DTS console
Log on to the DTS console.
In the left-side navigation pane, click Data Migration.
In the upper-left corner, select the region where the data migration instance resides.
DMS console
The actual steps may vary based on the DMS console mode and layout. For details, see Simple mode and Customize the layout and style of the DMS console.
Log on to the DMS console.DTS consoleDMS console
In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Data Migration.
From the drop-down list to the right of Data Migration Tasks, select the region where the data migration instance resides.
Step 2: Create a task
Click Create Task to go to the task configuration page.
Step 3: Configure source and destination databases
After configuring the source and destination databases, read the Limits displayed at the top of the page. Skipping this step may cause the task to fail or result in data inconsistency.
Source database parameters:
| Parameter | Description |
|---|---|
| Task Name | A name for the DTS task. DTS generates a name automatically. Specify a descriptive name to make the task easy to identify. The name does not need to be unique. |
| Select Existing Connection | If the database instance is registered with DTS, select it from the drop-down list. DTS automatically fills in the connection parameters. See Manage database connections. If not, configure the parameters below. |
| Database Type | Select MongoDB. |
| Access Method | Select the connection type based on where the source database is deployed. This example uses Public IP Address. For other connection types, complete the required preparations first. |
| Instance Region | Select the region where the self-managed MongoDB database resides. If the exact region is not listed, select the closest available region. |
| Architecture | Select Replica Set. |
| Migration Method | The method DTS uses to capture incremental changes from the source database. Options: Oplog (recommended) and ChangeStreamChange Streams. See the note below for guidance. |
| Endpoint Type | Select Standalone or Multi-node. This parameter appears only when the access method is Express Connect, VPN Gateway, or Smart Access Gateway, Public IP Address, or Cloud Enterprise Network (CEN). |
| Domain Name or IP | The domain name or IP address of the source database. This parameter appears only when Endpoint Type is Standalone. |
| Port Number | The service port of the source database. This parameter appears only when is . |
| Endpoint | The connection address of the source database in <IP>:<Port> format. Use a domain name that can be resolved over the Internet. Press Enter to separate multiple addresses. This parameter appears only when is Multi-node. |
| Authentication Database | The database that the account belongs to. If unchanged, enter admin. |
| Database Account | The account for the self-managed MongoDB database. For required permissions, see . If Access Method is Self-managed Database on ECS or Database Gateway, enter the shard node account. |
| Database Password | The password for the database account. |
| Encryption | Whether to encrypt the connection. Options: Non-encrypted, SSL-encrypted, or Mongo Atlas SSL. Available options depend on the and Architecture selections. If you select Replica Set architecture, is not Alibaba Cloud Instance, and Encryption is SSL-encrypted, you can upload a CA certificate to verify the connection. |
Choose a migration method based on your setup:
Oplog (recommended): Available when the oplog feature is enabled on the source. By default, oplog is enabled for both self-managed MongoDB databases and ApsaraDB for MongoDB instances. Oplog provides lower latency due to faster log pulling.
ChangeStream: Available when change streams are enabled on the source. See Change Streams. Required for inelastic Amazon DocumentDB clusters. If you select Sharded Cluster architecture, you do not need to configure Shard account and Shard password.
Destination database parameters:
| Parameter | Description |
|---|---|
| Select Existing Connection | If the database instance is registered with DTS, select it from the drop-down list. DTS automatically fills in the connection parameters. See Manage database connections. If not, configure the parameters below. |
| Database Type | Select MongoDB. |
| Select Alibaba Cloud Instance. | |
| Instance Region | Select the region where the destination ApsaraDB for MongoDB instance resides. |
| Replicate Data Across Alibaba Cloud Accounts | Select No if the destination instance belongs to the same Alibaba Cloud account. |
| Select the architecture of the destination ApsaraDB for MongoDB instance. | |
| Instance ID | Select the ID of the destination instance. |
| Authentication Database | The database that the account belongs to. If unchanged, enter admin. |
| Database Name | The name of the database in the destination instance that receives the migrated data. |
| Database Account | The account for the destination ApsaraDB for MongoDB instance. For required permissions, see . |
| Database Password | The password for the database account. |
| Whether to encrypt the connection. Options: Non-encrypted, , or Mongo Atlas SSL. If the destination is an ApsaraDB for MongoDB instance with Sharded Cluster architecture, is not available. |
Step 4: Test connectivity
Click Test Connectivity and Proceed, then click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.
Make sure the CIDR blocks of DTS servers are added to the security settings of both the source and destination databases. You can add them automatically or manually. See Add DTS server IP addresses to a whitelist.
Step 5: Configure migration objects
On the Configure Objects page, configure the following:
| Parameter | Description |
|---|---|
| Migration Types | Select the migration types to run. To perform full migration only, select Schema Migration and Full Data Migration. To migrate without downtime, also select Incremental Data Migration. If you skip Schema Migration, create the target database and collections manually and enable object name mapping in Selected Objects. If you skip , stop writes to the source database during the migration to keep data consistent. |
| Processing Mode of Conflicting Tables | Controls behavior when collection names conflict between source and destination. Precheck and Report Errors (default): fails the precheck if identical collection names are found—resolve conflicts before proceeding, or use object name mapping to rename collections. Ignore Errors and Proceed: skips the conflict check. Data consistency is not guaranteed—DTS skips records with duplicate primary keys, and migration may fail for some data. |
| Capitalization of Object Names in Destination Instance | The capitalization policy for database and collection names in the destination. Defaults to DTS default policy. Change this to match the source or destination if needed. See Specify the capitalization of object names in the destination instance. |
| Source Objects | Select the databases or collections to migrate, then click the arrow icon to move them to Selected Objects. |
| Right-click an object to rename it in the destination or specify the target object. To remove an object, click it and then click the remove icon. Right-click to select incremental migration mode per object, or to specify WHERE conditions for data filtering (full migration only). See Specify filter conditions. Using object name mapping for databases or collections may prevent dependent objects from migrating. |
Click Next: Advanced Settings and configure the following:
| Parameter | Description |
|---|---|
| Dedicated Cluster for Task Scheduling | By default, DTS schedules the task to the shared cluster. Purchase a dedicated cluster to isolate workloads. See What is a DTS dedicated cluster? |
| Retry Time for Failed Connections | How long DTS retries after a connection failure. Range: 10–1,440 minutes. Default: 720. Set to at least 30 minutes. If DTS reconnects within this window, the task resumes; otherwise, it fails. If multiple tasks share the same source or destination, the most recently set value takes effect. DTS charges during retry. |
| Retry Time for Other Issues | How long DTS retries after DDL or DML operation failures. Range: 1–1,440 minutes. Default: 10. Set to at least 10 minutes. Must be less than . |
| Enable Throttling for Full Data Migration | Limits the read and write rate during full migration to reduce server load. Configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s). Available only when is selected. |
| Only one data type for primary key _id in a table of the data to be synchronized | Controls how DTS handles _id field data types during full migration. Yesalert notification settings: DTS skips scanning data types and migrates only one data type per collection. : DTS scans all data types and migrates all of them. Enable this based on your data model—incorrect settings may cause data loss. Available only when is selected. |
| Enable Throttling for Incremental Data Migration | Limits the rate during incremental migration. Configure RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s). Available only when is selected. |
| Environment Tag | A tag to identify the DTS instance. Optional. |
| Configure ETL | Whether to apply extract, transform, and load (ETL) transformations. Yes: Enter data processing statements in the code editor. See Configure ETL in a data migration or data synchronization task. : No ETL transformations. See What is ETL? |
| Monitoring and Alerting | Whether to send alerts when the task fails or migration latency exceeds a threshold. : Configure the alert threshold and notification settings. See Configure monitoring and alerting when you create a DTS task. : No alerting. |
Click Next Step: Data Verification to set up data verification. See Configure a data verification task.
Step 6: Save settings and run the precheck
To preview the API parameters for this configuration, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.
Click Next: Save Task Settings and Precheck to save the task and start the precheck.
DTS runs a precheck before starting the migration. The task cannot start until the precheck passes.
If a check item fails, click View Details next to the failed item, fix the issue based on the results, and run the precheck again.
If a check item triggers an alert:
If the alert cannot be ignored, click View Details, fix the issue, and rerun the precheck.
If the alert can be ignored, click Confirm Alert Details, click Ignore in the dialog box, and click OK. Then click Precheck Again. Ignored alerts may lead to data inconsistency.
Step 7: Purchase an instance
Wait until Success Rate reaches 100%, then click Next: Purchase Instance.
On the Purchase Instance page, configure the following:
Parameter Description Resource Group The resource group for the migration instance. Default: default resource group. See What is Resource Management? Instance Class The instance class determines migration speed. Select based on your workload. See Instance classes of data migration instances. Read and agree to the Data Transmission Service (Pay-as-you-go) Service Terms by selecting the check box.
Click Buy and Start. In the dialog box, click .
Monitor the task on the page.
Full migration only: The task stops automatically when complete. The status shows Completed.
Full + incremental migration: The task runs continuously and never stops automatically. The status shows Running.