Migrate data from MongoDB to document database (MongoDB-compatible) - PolarDB

This topic describes how to use Alibaba Cloud Data Transmission Service (DTS) to smoothly migrate your MongoDB database to the document database feature of a PolarDB for MySQL cluster. This topic also provides an optional remote sync solution to facilitate business rollback.

Migration solution overview

Migration architecture

This migration uses Data Transmission Service (DTS) to synchronize data from the source MongoDB instance to the destination PolarDB cluster.

Forward synchronization (MongoDB to PolarDB): This process is performed using DTS. DTS first performs a full migration to copy all existing data from the source database. Then, DTS starts an incremental migration to capture and apply Oplog or Change Stream events from the source MongoDB instance. This synchronizes data changes in real time and ensures eventual data consistency between the source database and the destination PolarDB cluster.
Remote sync (optional, PolarDB to MongoDB): To facilitate a quick rollback in case of a business cutover failure, you can configure an optional remote data synchronization link from the PolarDB cluster to the source MongoDB instance. This link uses the binary logging (binlog) of PolarDB and Realtime Compute for Apache Flink.

Migration impact

Performance: Data migration, especially during the full migration phase, imposes an I/O load on the source and destination databases. We recommend that you perform the migration during off-peak hours.
Cost: The DTS data migration task and the Realtime Compute for Flink instance used for remote sync incur fees. After the migration is complete, release the resources that are no longer needed to avoid unnecessary costs.

Preparations

Before you start the migration, complete the following preparations to ensure that all prerequisites are met.

Category	Check item	Instructions and notes
Version and features	Destination PolarDB cluster	The cluster version must be MySQL 8.0.2 and have the document database compatibility feature enabled. (Optional) Parameter settings: To configure remote sync, you must enable binary logging (binlog) for the PolarDB cluster. Because enabling this feature requires a cluster restart, complete this setting before you start the migration process.
Version and features	Source MongoDB instance	Incremental migration depends on the Oplog or Change Stream of the source MongoDB database. Ensure that the source database is a replica set or a sharded cluster and has enough Oplog space to cover the entire duration of the full migration.
Network planning	Network environment	The source MongoDB instance, the destination PolarDB cluster, and the Flink instance to be created must be in the same virtual private cloud (VPC).
Accounts and Permissions	Source MongoDB instance	Prepare a database account for DTS migration. This account must have read permissions on all data to be migrated.
Accounts and Permissions	Destination PolarDB cluster	Create a dedicated account for the document database. Before the business cutover, control the write permissions on the destination PolarDB cluster. Allow only the account used by DTS to write data to prevent data contamination.

Migration steps

Step 1: Create and configure a DTS migration task

Configure the data migration link from the source MongoDB to the destination PolarDB for MySQL. This topic uses a replica set instance of ApsaraDB for MongoDB as an example.

Log on to the Data Transmission Service (DTS) console. In the navigation pane on the left, click Data Migration. In the upper-left corner of the page, select the region where your instance is located.
Click Create Task to go to the task configuration page.

Configure the source and destination databases.

Category	Configuration	Description
None	Task Name	DTS automatically generates a task name. Specify a descriptive name for easy identification. The name does not have to be unique.
Source Database	Database Type	Select a type based on where your source database is deployed. This topic uses NoSQL database > MongoDB as an example. Note If the source or destination database is hosted on another cloud platform, you must perform other preparations.
	Access Method	Select Alibaba Cloud Instance. Note You can adjust this setting as needed.
	Instance Region	Select the region where the MongoDB instance is located. Note If the source or destination database is hosted on another cloud platform, you can select a nearby region.
	Replicate Data Across Alibaba Cloud Accounts	Adjust this setting as needed. This topic uses No (same account) as an example. Note You can configure data migration or synchronization for instances under different Alibaba Cloud accounts.
	Architecture	Adjust this setting as needed. This topic uses Replica Set as an example.
	Migration Method	Select Oplog.
	Instance ID	Select the MongoDB instance under your account.
	Authentication Database	Enter the Authentication Database.
	Database Account	Enter the account for the MongoDB instance.
	Database Password	Enter the password for the MongoDB instance.
	Encryption	Based on your MongoDB instance, select Non-encrypted or SSL-encrypted.
Destination Database	Database Type	This is fixed to NoSQL database > MongoDB.
	Access Method	Select Express Connect, VPN Gateway, or Smart Access Gateway.
	Instance Region	Select the region where the PolarDB cluster is located.
	Connected VPC	Select the VPC of the PolarDB cluster.
	Architecture	This is fixed to Replica Set. Sharded Cluster is not supported.
	Endpoint Type	This is fixed to Standalone.
	Domain Name or IP	Enter the private endpoint of the document database.
	Port Number	Enter the port that corresponds to the private endpoint of the document database. The default port is 3306.
	Authentication Database	Enter `admin`.
	Database Account	Enter the dedicated account for the document database. This is fixed to `doc_root`.
	Database Password	Enter the password for the dedicated account for the document database.
	Encryption	Select Non-encrypted.
	Primary/Secondary vSwitch in VPC Tunnel	Select the VPC data channel that DTS uses to access your data source: vSwitch (Primary): Select the vSwitch to which the PolarDB cluster belongs. vSwitch (Secondary): You can select one as needed.

Click Test Connectivity and Proceed. In the CIDR Blocks of DTS Servers dialog box, add the IP address of the Destination database to the whitelist of the PolarDB cluster. Click Confirm authorization and test link. If the connection test is successful, you are automatically redirected to the next step to configure the task objects.
Warning
Adding the public CIDR blocks of DTS servers, whether automatically or manually, may introduce security risks. By using this product, you acknowledge and accept these potential risks. You are responsible for implementing basic security measures, including but not limited to using strong passwords, restricting open ports, using authentication for internal API calls, regularly reviewing and restricting unnecessary network segments, or connecting through private networks such as Express Connect, VPN Gateway, or Smart Access Gateway.

Configure the task objects.

Object configuration

Configuration	Description
Migration Types	Select Schema Migration, Full Data Migration, and Incremental Data Migration. Note Incremental Data Migration does not support databases created after the task starts. For more information, see Migration types.
Handling of Existing Tables in Destination	Precheck and Report Errors: Checks whether the destination is empty. If the destination to be migrated to is empty, the precheck is passed. If not, an error is reported during the precheck, and the data migration job does not start. Ignore Errors and Proceed: Skips the Check the existence of objects in the destination database. check item. Warning If you select Ignore Errors and Proceed and a key in the destination has the same name as a key in the source during migration, the data from the source overwrites the data in the destination. This may cause data loss in the destination. Proceed with caution.
Migration Objects	In the Source Objects box, click the objects to migrate, and then click to move them to the Selected Objects box.

Click Advanced Settings and configure the parameters as needed.
Click Data Verification and configure the parameters as needed.
Note
To perform data validation, you can select Verify based on the number of table rows for Full Data Verification.
Perform a precheck. In the lower-right corner of the page, click Next: Save Task Settings and Precheck and wait for the precheck to complete. A precheck is performed before the migration task starts. The migration task can start only after it passes the precheck.
Note
If the precheck fails, click the icon next to the failed check item to view the details.
- You can fix the issue as prompted and run the precheck again.
- If you do not need to fix the issues that trigger alerts, you can select Ignore and run the precheck again.
Purchase a DTS instance:
1. On the purchase page, select a Instance Class and select Data Transmission Service (Pay-as-you-go) Service Terms.
2. Click Buy and Start. The migration task starts.

Step 2: Monitor the migration task and validate data

After the task starts, you can monitor its status in the DTS console. The task proceeds through the schema migration, full data migration, and incremental data migration phases.

Monitor migration progress:
- Full migration: Observe the data volume migration progress.
- Incremental migration: Monitor the Latency metric for Incremental Write. When the full migration is complete and the Incremental Write latency is stable and within a few seconds, it indicates that the destination PolarDB cluster is nearly in sync with the source database.
View migration performance:
- On the Performance Monitoring page, you can view the Performance of Full Data Migration and Performance of Incremental Data Migration.

Step 3: (Optional) Configure remote sync

To ensure a seamless business cutover, you can configure a remote sync link from the PolarDB cluster to the source MongoDB instance before the cutover. After your business switches to PolarDB, this link synchronizes data changes from the PolarDB cluster back to the source MongoDB instance in real time. This provides the data required for potential rollback operations. The configuration process is as follows:

Environment preparation

Ensure that you have enabled binlog for the PolarDB cluster.
(Optional) Create a privileged database account: If you have not created a privileged account, go to the PolarDB console. On the Settings and Management > Accounts page of the cluster, create a privileged account.
Activate and configure Flink: Activate Realtime Compute for Apache Flink and create a Flink workspace.
Important
The Flink workspace must be in the same VPC as the PolarDB cluster.
Configure the IP address whitelist for the PolarDB cluster:
1. In the Flink console, click Details for the workspace. On the Workspace Details page, obtain its CIDR block.
2. Go to the PolarDB console. On the Settings and Management > Cluster Whitelists page of the cluster, Add Whitelist and add the Flink CIDR block to it.
3. Go to the MongoDB console. On the Data Security > Whitelist Settings page of the cluster, add the Flink CIDR block.
Verify the connectivity between the PolarDB cluster and the Flink workspace:
1. In the Flink console, go to the workspace and click the Network Probe icon in the upper-right corner.
2. Enter the private endpoint and port of the primary node of the PolarDB cluster, and then click Probe.
3. A dialog box appears with the message Network probe successful. This indicates that the cluster whitelist is configured correctly.

Deploy the Flink job

Download the remote sync tool: PolarDBBackSync.jar.

Prepare the configuration file: Create a configuration file named application.yaml with the following content:

source:
    # Private endpoint of the PolarDB primary node
    hostname: pc-xxx.mysql.polardb.rds.aliyuncs.com
    # Private port of the PolarDB primary node
    port: 3306

# (Optional)
# whiteTableSet: Tables to be synchronized back. blackTableSet: Tables not to be synchronized back.
# You can declare only one of these two sets, or neither. If neither is declared, all tables are synchronized back.
filter:
  whiteTableSet:
  blackTableSet:

# Checkpoint interval in milliseconds. For example, 3s.
checkpoint:
  interval: 3000

Upload files: Go to the Flink console and go to the destination workspace. On the File Management page, upload PolarDBBackSync.jar and application.yaml.

Store credentials securely: To avoid exposing keys in plaintext, use the Variable Management feature of Flink to store sensitive information. On the Variable Management page, add the following four variables:

Variable Name	Variable Value
`polardbusername`	The privileged database account for PolarDB.
`polardbpassword`	The password for the privileged database account of PolarDB.
`mongodbendpoint`	The endpoint of MongoDB. We recommend that you use the ConnectionStringURI (high-availability endpoint).

Deploy and start the job:

Go to the Job O&M page and select Deploy Job > JAR Job.

Configure the following main parameters. You can configure other parameters based on your business environment. Then, click Deploy.

Parameter Name	Reference
Deployment Mode	This is fixed to Streaming Mode.
Deployment Name	Enter a name for the job deployment. This topic uses PolarDBBackSync as an example.
Engine Version	This is fixed to vvr-11.3-jdk11-flink-1.20.
JAR URI	Select the uploaded `PolarDBBackSync.jar` file.
Entry Point Class	This is fixed to `org.example.PolarDBToMongoDBCdcJob`.
Entry Point Main Arguments	This is fixed to: `--polardbusername ${secret_values.polardbusername}` `--polardbpassword ${secret_values.polardbpassword}` `--mongodbendpoint ${secret_values.mongodbendpoint}`
Additional Dependencies	Select the uploaded `application.yaml` file.

After the deployment is successful, click Start > Stateless Start. Wait for the job status to become Running.

Step 4: Perform the business cutover

When the incremental synchronization latency of DTS is stable and low, you can start the business cutover. Perform the cutover during off-peak hours.

Final validation: Perform sample comparisons of critical business data or conduct a final check on the row counts of core tables.
Stop source writes: Stop all business applications or services that write data to the source MongoDB instance.
Stop the DTS task: In the DTS console, Pause Task. Do not Stop Task or Delete Task.
Switch application connection: Change the database connection configuration of your business applications from the source MongoDB instance address to the document database address of PolarDB.
(Optional) Start the remote sync task: When you cut over your business to PolarDB, start the remote sync task. This task uses Realtime Compute for Apache Flink to synchronize data back to the source to provide a safeguard for business rollback.
Start the business: Restart your business applications and verify that core functionalities and the business run as expected on PolarDB.
(Optional) Stop the remote sync task: After the cutover is complete, the business has been running stably for a period of time, and you confirm that data consistency meets your business requirements, you can safely stop the remote sync task (stop the Flink job and release related resources).
Observation phase:
- Closely monitor business logs and database performance metrics to confirm that there are no abnormalities.
- If everything is normal after a period of stable operation, the cutover is successful.

Step 5: Clean up resources

After the migration and cutover are successful, clean up the resources that are no longer needed to avoid unnecessary costs.

Data Transmission Service (DTS): In the DTS console, Delete Task.
Realtime Compute for Apache Flink:
1. Stop the job: Go to the Flink console. In the destination workspace, go to the Job O&M page, find the target job, and click Stop.
2. Release the instance: Return to the Flink console, find the destination workspace, and click Release Resources.
PolarDB cluster: Depending on your business needs, you can choose to shorten the binlog retention period or disable binlog.
Note
Because disabling the binlog feature requires a cluster restart, perform this operation during off-peak hours and ensure that your applications have a reconnection mechanism.