Data Transmission Service:Migrate data from ApsaraDB for MongoDB to AnalyticDB for PostgreSQL

Source database

• Bandwidth: The source database server must have sufficient outbound bandwidth, or migration speed is affected.

• Collection limit: A single migration task can migrate a maximum of 1,000 collections when name mapping is configured. If this limit is exceeded, split the objects across multiple tasks.

• Full migration only: Standalone MongoDB instances, Azure Cosmos DB for MongoDB, and Amazon DocumentDB elastic clusters support full data migration only.

• Incremental migration — oplog/change streams retention: The source database must have the oplog enabled and retained for at least 7 days, or have change streams enabled with at least 7 days of subscribable history. If this requirement is not met, the task may fail or data loss may occur. Such issues are not covered by the DTS Service-Level Agreement (SLA).

  • Use the oplog when possible — it provides lower latency for incremental tasks.

  • Change streams require MongoDB 4.0 or later.

  • For Amazon DocumentDB (non-elastic cluster), you must manually enable change streams, set Migration Method to ChangeStreamChange Streams, and set Architecture to Sharded Cluster.

• Sharded cluster — unique `_id`: The _id field in all collections to be migrated must be unique. Otherwise, data inconsistency may occur.

• Sharded cluster — Mongos node count: The number of Mongos nodes cannot exceed 10.

• Sharded cluster — orphaned documents: Ensure there are no orphaned documents in the cluster. Otherwise, data inconsistency or task failure may occur. See Orphaned Document and How do I purge orphaned documents from a MongoDB sharded cluster?.

• Self-managed sharded cluster — connection type: Only Public IP, Leased Line/VPN Gateway/Smart Gateway, and CEN are supported.

• Self-managed sharded cluster — MongoDB 8.0+ with Oplog: The shard account used by the migration task must have the directShardOperations permission. Grant it by running:

  db.adminCommand({ grantRolesToUser: "username", roles: [{ role: "directShardOperations", db: "admin"}]})

  Replace username with the actual shard account name.

• SRV records: Connecting to MongoDB using an SRV record is not supported.

• Sharded cluster with active balancer: If the balancer is rebalancing data during migration, instance latency may occur.

Operations during migration

• During full data migration, do not change database or collection schemas, including updates to array-type data.

• If performing full migration only (no incremental migration), do not write new data to the source instance during migration.

• For sharded cluster sources, do not run commands that change data distribution — such as shardCollection, reshardCollection, unshardCollection, moveCollection, or movePrimary — while the migration instance is running.

Other limitations

• Only collection-level migration is supported.

• The admin, config, and local databases cannot be migrated.

• Transaction information is not retained. Transactions from the source are converted to individual records in the destination.

• FLOAT columns are migrated with a precision of 38 digits; DOUBLE columns with a precision of 308 digits. DTS reads these values using ROUND(COLUMN,PRECISION). Verify that this precision meets your business requirements before migrating.

• Full data migration uses concurrent INSERT operations, which can cause fragmentation. After migration, the disk space used by destination tables may exceed that of the source collections.

• DTS attempts to resume failed tasks within 7 days. Before switching business traffic to the destination, end or release the task — or revoke the destination account's write permissions using the revoke command — to prevent the source from overwriting destination data after an automatic resume.

• DTS calculates latency based on the timestamp of the last migrated record compared to the current time. If the source has not been updated for a long time, the latency figure may be inaccurate. To refresh it, perform an update on the source database.

• Migrating time-series collections (introduced in MongoDB 5.0) is not supported.

• If a task fails, DTS support staff will attempt to restore it within 8 hours. During restoration, the task may be restarted or its parameters adjusted. Only DTS task parameters are modified — database parameters are not changed. Parameters that may be adjusted are listed in Modify instance parameters.

Step 1: Go to the migration task list

Use either the DTS console or the DMS console.

Step 2: Create a task

Click Create Task to open the task configuration page.

Step 3: Configure source and destination databases

After completing the configuration, click Test Connectivity and Proceed.

Ensure that the IP address ranges of the DTS service are automatically or manually added to the security settings of the source and destination databases to allow access from DTS servers. For self-managed databases (where Access Method is not Alibaba Cloud Instance), also click Test Connectivity in the CIDR Blocks of DTS Servers dialog. See Add DTS server IP addresses to a whitelist.

Step 4: Configure migration objects

On the Configure Objects page, set the following:

In Selected Objects, configure object name mapping and field mappings:

Map database names to schemas:

1. Right-click the database in Selected Objects.

2. Change Database Name to the name of the target schema in AnalyticDB for PostgreSQL.

3. (Optional) In Select DDL and DML Operations to Be Synchronized, select operations for incremental migration.

4. Click OK.

Map collection names to tables:

1. Right-click the collection in Selected Objects.

2. Change Table Name to the name of the target table in AnalyticDB for PostgreSQL.

3. (Optional) Set filter conditions. See Set filter conditions.

4. (Optional) In Select DDL and DML Operations to Be Synchronized, select operations for incremental migration.

Configure field mappings:

DTS maps collection fields to destination columns automatically. Review the bson_value() expressions and configure Column Name, Type, Length, and Precision for each field.

Click OK.

Step 5: Configure advanced settings

Click Next: Advanced Settings and configure:

Step 6: Save settings and run a precheck

Click Next: Save Task Settings and Precheck.

To view OpenAPI parameters for this configuration, hover over the Next: Save Task Settings and Precheck button and click Preview OpenAPI parameters.

DTS runs a precheck before starting the task. If the precheck fails:

• Click View Details next to the failed item, fix the issue, and click Precheck Again.

• For warnings on items that can be ignored: click Confirm Alert Details, Ignore, OK, and Precheck Again. Ignoring warnings may cause data inconsistency.

Step 7: Purchase the instance

1. When Success Rate reaches 100%, click Next: Purchase Instance.

2. On the Purchase page, configure the instance:

   Parameter	Description
   Resource Group Settings	Select the resource group for the instance. Default: default resource group. See What is Resource Management?.
   Instance Class	Select a link specification based on your required migration speed. Higher specifications support faster migration. See Data migration link specifications.

3. Read and accept Data Transmission Service (Pay-as-you-go) Service Terms.

4. Click Buy and Start. In the confirmation dialog, click OK.

The task appears on the Data Migration Tasks list page. Monitor its progress there.

For full-migration-only tasks, the task stops automatically after full migration completes. The Status changes to Completed.

For tasks that include incremental migration, the task continues running. The Status remains Running until you stop it manually.

Source MongoDB document

{
  "_id": "62cd344c85c1ea6a2a9f****",
  "person": {
    "name": "neo",
    "age": 26,
    "sex": "male"
  }
}

Destination AnalyticDB for PostgreSQL table schema

Data in the destination table after migration

The following shows what the destination table looks like after migration, given the source document above:

Note that sex is not migrated in this example because no column was configured for it.

Field mapping configuration