Migrate a self-managed replica set database to ApsaraDB for MongoDB by using DTS - ApsaraDB for MongoDB

Combine full and incremental data migration to move your self-managed MongoDB replica set data to ApsaraDB for MongoDB with zero downtime.

You can also use MongoDB tools to migrate a self-managed database to a replica set instance. Data migration and synchronization covers additional scenarios.

Prerequisites

The source and destination MongoDB versions must be compatible. Check the Migration solutions overview.ta#concept_26618_zh" baseUrl="t6682_v1_36_5.dita" data-node="17092">.
The destination ApsaraDB for MongoDB instance should have at least 10% more storage than the source database.raDB for MongoDB instance is 10% larger than that used by the self-managed MongoDB database.

Usage notes

Full data migration consumes resources on both the source and destination, which increases server load and may cause service disruptions. Migrate during off-peak hours if your database handles high traffic or runs on limited hardware.nal database. Do not migrate it unless necessary.
When DTS performs full data migration, it consumes resources of the source and destination databases, which may cause the database server load to rise. If the database workload is heavy or the server specifications are low, the database pressure may increase and the database service may become unavailable. Evaluate the impact carefully before migration, and perform data migration during off-peak hours.
e database server load to rise. If the database workload is heavy or the server specifications are low, the database pressure may increase and the database service may become unavailable. Evaluate the impact carefully before migration, and perform data migration during off-peak hours.
Check Versions and storage engines for supported MongoDB configurations. Verify compatibility before migrating across versions or storage engines.="t6682_v1_36_5.dita" data-node="6653">. If you need to perform cross-version or cross-engine migration, confirm the compatibility in advance.
The admin and local databases cannot be used as source or destination databases.
cannot be used as source or destination databases.
DTS writes data to the destination concurrently, which may cause the destination to use 5%–10% more storage than the source.e space occupied by the destination is 5% to 10% larger than that of the source.
Ensure no documents in the destination share a primary key (_id by default) with documents in the source. Duplicate keys may cause data to be overwritten. Delete duplicates in the destination before migration if doing so does not affect your business.Otherwise, data loss may occur. If the destination has the same primary key as the source, clear the related data in the destination (delete documents with the same _id as the source) without affecting your business.

Billing

Migration type	Link configuration fee	Internet traffic fee
Full data migration	Free of charge.	Charged only when data is migrated out of Alibaba Cloud over the Internet. For more information, see Billing overview.
Incremental data migration	Charged. For more information, see Billing overview.

Migration types

Full data migration: Migrates all existing data from the selected objects in the source MongoDB database to the destination MongoDB database.
Note
This includes databases, collections, and indexes.
"note_wtw_n1y_f19">
Supports migration of database, collection, and index.
Incremental data migration: After the full data migration is complete, DTS synchronizes data changes from the source MongoDB database to the destination.
Note
- Synchronizes CREATE and DELETE operations for databases, collections, and indexes.
- Synchronizes INSERT, UPDATE, and DELETE operations for documents.
s for databases, collections, and indexes. operations on database, collection, and index.
Synchronizes INSERT, UPDATE, and DELETE operations for documents.nsert, delete, and update operations on documents.

Database account permissions

Migration data source	Full data migration	Incremental data migration
Self-managed MongoDB database	The read permission on the database to be migrated	The read permission on the database to be migrated, the admin database, and the local database
ApsaraDB for MongoDB database	The readWrite permission on the destination database	The readWrite permission on the destination database

To create accounts and grant permissions:

ApsaraDB for MongoDB: Manage MongoDB database users in DMSse DMS to manage MongoDB database users.
Self-managed MongoDB: db.createUser() (official MongoDB documentation)oDB Create User documentation.

Configure and run a migration task

Step 1: Go to the Data Migration page

Via the DTS console:

Log on to the DTS console.
In the left-side navigation pane, click Data Migration.
In the upper-left corner, select the region where the migration instance resides.

Via the DMS console:

The actual steps may vary based on the DMS console mode and layout. See Simple mode and Customize the layout and style of the DMS console.

Log on to the DMS console.
In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Data Migration.
From the drop-down list to the right of Data Migration Tasks, select the region where the migration instance resides.

Step 2: Create the task and configure source and destination databases

Click Create Task.
Configure the source and destination databases using the parameters in the following tables.

Warning

After configuring the source and destination databases, review the Limits displayed at the top of the page. Skipping this step may cause the task to fail or result in data inconsistency.

Source database parameters

Parameter	Description
Task Name	The task name. DTS generates a name automatically. Specify a descriptive name to identify the task easily. Task names do not need to be unique.
Select Existing Connection	If the source database is registered with DTS, select it from the drop-down list. DTS populates the remaining parameters automatically. Otherwise, configure the parameters below manually.
Database Type	Select MongoDB.
Access Method	Select the connection type based on the location of the source database. This topic uses Public IP Address as an example. If you select a different connection type, complete the required preparations.
Instance Region	Select the region where the source database resides. If the region is not listed, select the geographically closest region.
Architecture	Select Replica Set.
Migration Method	Select the method for migrating incremental data: Oplog (recommended) or ChangeStream. Oplog is available when the oplog feature is enabled on the source. ChangeStream is available when change streams are enabled. If the source is an inelastic Amazon DocumentDB cluster, you must select ChangeStream and set the Architecture parameter to Sharded Cluster. If you select Sharded Cluster for the Architecture parameter, you do not need to configure the Shard account and Shard password parameters.
Endpoint Type	Select Standalone or Multi-node. Available when Access Method is Express Connect, VPN Gateway, or Smart Access Gateway, Public IP Address, or Cloud Enterprise Network (CEN).
Domain Name or IP	The domain name or IP address of the source database. Available when Endpoint Type is Standalone.
Port Number	The service port of the source database. Available when Endpoint Type is Standalone.
Endpoint	The endpoint of the source database in `<IP>:<Port>` format. Press Enter to add multiple endpoints. Available when Endpoint Type is Multi-node.
Authentication Database	The database that the source account belongs to. Default: `admin`.
Database Account	The account used to connect to the source database. For required permissions, see Permissions required. If Access Method is Self-managed Database on ECS or Database Gateway, enter the account of the shard node.
Database Password	The password for the database account.
Encryption	The connection encryption method: Non-encrypted, SSL-encrypted, or Mongo Atlas SSL. Available options depend on Access Method and Architecture. SSL-encrypted is unavailable when Architecture is Sharded Cluster and Migration Method is Oplog. If Architecture is Replica Set, Access Method is not Alibaba Cloud Instance, and Encryption is SSL-encrypted, upload a CA certificate to verify the connection.

Destination database parameters

Parameter	Description
Select Existing Connection	If the destination instance is registered with DTS, select it from the drop-down list. Otherwise, configure the parameters below manually.
Database Type	Select MongoDB.
Access Method	Select Alibaba Cloud Instance.
Instance Region	Select the region where the destination ApsaraDB for MongoDB instance resides.
Replicate Data Across Alibaba Cloud Accounts	Select No to use an instance in the current account.
Architecture	Select the architecture of the destination instance.
Instance ID	Select the ID of the destination instance.
Authentication Database	The database that the destination account belongs to. Default: `admin`.
Database Name	The name of the database in the destination instance where migrated objects will be stored.
Database Account	The account used to connect to the destination instance. For required permissions, see Permissions required.
Database Password	The password for the database account.
Encryption	The connection encryption method. SSL-encrypted is unavailable when the destination uses a sharded cluster architecture.

Step 3: Test connectivity

Click Test Connectivity and Proceed, then click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Make sure the CIDR blocks of DTS servers are added to the allowlists of the source and destination databases. See Add DTS server IP addresses to a whitelist.

Step 4: Configure migration objects

On the Configure Objects page, configure the migration objects using the following parameters.

Parameter	Description
Migration Types	Select the migration types: Schema Migration, Full Data Migration, and optionally Incremental Data Migration. For zero-downtime migration, select all three. If you skip Schema Migration, create the target databases and collections in the destination before starting. If you skip Incremental Data Migration, do not write to the source during migration.
Processing Mode of Conflicting Tables	Precheck and Report Errors: fails the precheck if the destination contains collections with the same names as the source. Use object name mapping to rename migrated collections if existing collections cannot be deleted. Ignore Errors and Proceed: skips the precheck. DTS does not migrate records with the same primary keys as existing destination records. Data consistency is not guaranteed.
Capitalization of object names in destination instance	The capitalization policy for database and collection names in the destination. Default: DTS default policy. See Capitalization policy for object names in the destination database.
Source Objects	Select databases or collections to migrate. Click the icon to add them to Selected Objects.
Selected Objects	Right-click an object to rename it in the destination (object name mapping), set the incremental migration mode, or specify WHERE conditions for full migration data filtering. To remove an object, click it and then click the icon.

Step 5: Configure advanced settings

Click Next: Advanced Settings and configure the following parameters.

Parameter	Description
Dedicated Cluster for Task Scheduling	By default, DTS schedules tasks on a shared cluster. Purchase a dedicated cluster for isolated resources. See What is a DTS dedicated cluster?
Retry Time for Failed Connections	The retry duration for connection failures. Valid values: 10–1,440 minutes. Default: 720 minutes. Set this to at least 30 minutes. If different tasks share the same source or destination database, the value set last takes precedence.
Retry Time for Other Issues	The retry duration for DDL or DML operation failures. Valid values: 1–1,440 minutes. Default: 10 minutes. Set this to at least 10 minutes. This value must be less than Retry Time for Failed Connections.
Enable Throttling for Full Data Migration	Limits read/write load on source and destination during full migration. Configure QPS to the source database, RPS of Full Data Migration, and data migration speed. Available only when Full Data Migration is selected.
Only one data type for primary key `_id` in a table of the data to be synchronized	Yes: DTS migrates only one data type of `_id` per collection without scanning all types. No: DTS scans and migrates all data types of `_id`. Enable this based on your data. Incorrect configuration may cause data loss. Available only when Full Data Migration is selected.
Enable Throttling for Incremental Data Migration	Limits load during incremental migration. Configure RPS of Incremental Data Migration and data migration speed. Available only when Incremental Data Migration is selected.
Environment Tag	An optional tag to identify the instance.
Configure ETL	Enable the extract, transform, and load (ETL) feature to transform data during migration. See Configure ETL in a data migration or data synchronization task.
Monitoring and Alerting	Configure alerts for task failure or high migration latency. If enabled, set the alert threshold and notification contacts. See Configure monitoring and alerting.

Step 6: Configure data verification (optional)

Click Next Data Verification to set up a data verification task. See Configure a data verification task.

Step 7: Run the precheck

Click Next: Save Task Settings and Precheck.

To preview the API parameters for this task configuration, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

DTS runs a precheck before starting the migration. The task starts only after passing the precheck.

If a check item fails, click View Details, resolve the issue, then click Precheck Again.
If an alert is triggered for a check item that can be ignored, click Confirm Alert Details > Ignore > OK > Precheck Again. Ignoring alerts may result in data inconsistency.

Step 8: Purchase and start the instance

Wait for Success Rate to reach 100%, then click Next: Purchase Instance.
On the Purchase Instance page, configure the following parameters.

Parameter	Description
Resource Group	The resource group for the migration instance. Default: default resource group. See What is resource management?
Instance Class	The instance class determines migration speed. Select based on your data volume and timing requirements. See Instance classes of data migration instances.

Read and select the checkbox to agree to Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start, then click OK in the confirmation dialog box.

What's next

Connect to an ApsaraDB for MongoDB replica set instance