Data Transmission Service:Migrate data from an ApsaraDB for MongoDB instance to a Lindorm instance

Step 1: Open the Data Migration page

Use one of the following methods:

Step 2: Configure source and destination databases

Click Create Task, then configure the parameters described in the following tables.

Source database parameters

Destination database parameters

Step 3: Test connectivity and proceed

Click Test Connectivity and Proceed at the bottom of the page.

DTS must be able to access both the source and destination databases. Make sure DTS server CIDR blocks are added to the security settings of both databases. See Add the CIDR blocks of DTS servers. If the source or destination is a self-managed database not accessed via Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Step 4: Configure objects to migrate

On the Configure Objects page, set the following parameters.

Migration types and objects

Parameter	Description
Migration Types	Select Full Data Migration for a one-time migration. Select both Full Data Migration and Incremental Data Migration to keep the source and destination in sync during migration and minimize downtime.
Processing Mode of Conflicting Tables	No configuration required.
Capitalization of Object Names in Destination Instance	The capitalization policy for database and collection names in the destination. Default: DTS default policy. See Specify the capitalization of object names in the destination instance.
Source Objects	Select the collections to migrate and click to move them to Selected Objects.

Configure selected objects

For each collection in Selected Objects, configure the schema name, table name, and column mappings.

The following example shows how a source MongoDB document maps to a Lindorm wide table row:

Source document (ApsaraDB for MongoDB)

{
  "_id": "62cd344c85c1ea6a2a9f****",
  "person": {
    "name": "neo",
    "age": "26",
    "sex": "male"
  }
}

Destination table row (Lindorm)

To configure the schema, table name, and column mappings:

1. Edit the schema name:

   1. In Selected Objects, right-click the database that contains the collections.

   2. In the Edit Schema dialog box, enter the database name to use in the Lindorm instance in the Schema Name field.

   3. (Optional) In Select DDL and DML Operations to Be Synchronized, select the operations to migrate during incremental data migration.

   4. Click OK.

2. Edit table names:

   1. In Selected Objects, right-click a collection.

   2. In the Edit Table Name dialog box, enter the table name to use in the Lindorm instance in the Table Name field.

   3. (Optional) Specify filter conditions. See Specify filter conditions.

   4. (Optional) In Select DDL and DML Operations to Be Synchronized, select the operations to migrate during incremental data migration.

3. Map source fields to destination columns: DTS automatically maps each field in the source collection using a bson_value() expression and assigns it to a column. In the bson_value() expression, the field name inside "" is the field name in the source MongoDB instance. For example, bson_value("age") maps the age field. (Optional) Click the icon next to a column to remove fields that do not need to be migrated. Check that the bson_value() expression matches the required destination columns:

   1. In the bson_value() expression in the Assign Value column, check the field name from MongoDB.

      The field within "" is the field name in MongoDB. For example, if the expression is bson_value("age"), this expression maps the age field in MongoDB.

   2. (Optional) Click the icon next to a column to remove fields that do not need to be migrated.

      Note

      To delete a field that you do not need to migrate, click in the row of the field.

   3. Configure the fields to be migrated.

      Perform the next steps based on whether the bson_value() expression meets your requirements.

      Fields for which the expression meets requirements

      1. Enter the Column Name for the destination table.

         Note

         • For SQL-created tables: use the column names defined in the table.

         • For Apache HBase API-created tables: use ROW for the primary key column, and Column family:Column name format (for example, person:name) for other columns. Create column mappings first. See Add column mappings for an Apache HBase API table.

      2. Select the Type for the column data.

         Important

         The data type of the destination column must be compatible with the source field type.

      3. (Optional) Set Length and Precision.

      4. Repeat these steps to map all required fields.

      Fields for which the expression does not meet requirements

      Note

      For example, fields that have a hierarchical structure (parent-child structure).

      1. Click the icon next to the column.

      2. Click + Add Column.

      3. Set Column Name, Type, Length, and Precision.

      4. In the text box under Assign Value, enter a bson_value() expression. For more information, see Value assignment example.

         Important

         • Always assign the primary key column as bson_value("_id").

         • For nested fields, specify both the field and subfield in the expression — for example, bson_value("person","name"). Using only bson_value("person") causes data loss for subfields.

      5. Repeat for each field.

4. Click OK.

Step 5: Configure advanced settings

Click Next: Advanced Settings and configure the following parameters.

Step 6: Run precheck and save settings

Click Next: Save Task Settings and Precheck.

To view the API parameters for this task configuration before saving, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

DTS runs a precheck before starting the migration. Address any precheck failures:

• For failed items: click View Details, fix the issue, then click Precheck Again.

• For alert items:

  • If the alert cannot be ignored, fix the issue and run the precheck again.

  • If the alert can be ignored, click Confirm Alert Details > Ignore > OK, then click Precheck Again. Ignoring alerts may result in data inconsistency.

Step 7: Purchase and start the instance

1. Wait until the Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the Purchase Instance page, configure the following parameters:

   Parameter	Description
   Resource Group	The resource group for the data migration instance. Default: default resource group. See What is Resource Management?
   Instance Class	The migration throughput tier. Higher classes provide faster migration. See Instance classes of data migration instances.

3. Read the Data Transmission Service (Pay-as-you-go) Service Terms and select the check box.

4. Click Buy and Start, then click OK in the confirmation dialog.

Verify migration status

On the Data Migration page, check the task status:

• Full data migration only: The task stops automatically when complete. The status changes to Completed.

• Full + incremental data migration: The incremental migration runs continuously without stopping. The status shows Running. Stop or release the task manually when you are ready to cut over to the destination database.

Source database limitations

• The source server must have sufficient outbound bandwidth. Insufficient bandwidth reduces migration speed.

• Collections must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, duplicate records may appear in the destination.

• DTS cannot migrate data from the admin, config, or local databases.

• DTS cannot connect to MongoDB over an SRV endpoint.

• If you select collections as migration objects and rename them, a single task can migrate up to 1,000 collections. Exceeding this limit causes a request error. Split the migration into multiple tasks.

Full data migration limitations

• Do not perform schema changes on databases or collections during full data migration, including updates to array types. Schema changes cause task failures or data inconsistency.

• Do not write data to the source database if you are running full data migration only. Writing to the source causes data inconsistency.

• Concurrent INSERT operations during full migration cause fragmentation in destination collections. The storage used by destination collections will be larger than in the source.

Incremental data migration limitations

• Only INSERT, UPDATE, and DELETE operations on documents are migrated. For UPDATE operations, only updates from $set are migrated.

• The oplog feature must be enabled and operation logs retained for at least 7 days, OR change streams must be enabled and DTS must be able to subscribe to changes from the last 7 days. If these requirements are not met, DTS may fail to obtain operation logs and the task may fail. In exceptional cases, data inconsistency or loss may occur.

• Use the oplog feature to obtain data changes when possible. Use change streams only if the source runs MongoDB V4.0 or later.

• Transaction information is not retained. Transactions are converted into a single record.

• DTS calculates migration latency based on the timestamp of the latest migrated record in the destination and the current timestamp in the source. If no updates occur on the source for an extended period, the latency value may be inaccurate. To refresh the latency, write an update to the source.

Sharded cluster limitations

• The _id field must be unique across all shards. Duplicate _id values cause data inconsistency.

• The number of Mongos nodes cannot exceed 10.

• Do not run the following commands on objects being migrated while the task is running: shardCollection, reshardCollection, unshardCollection, moveCollection, movePrimary. These commands change data distribution and cause data inconsistency.

• If the Balancer is active in the source sharded cluster, it may cause migration latency.

Lindorm destination limitations

• The destination Lindorm instance cannot contain collections with a column named _id or _value. If such columns exist, the task fails.

• Migrated data must comply with Limits on data requests.

• FLOAT precision defaults to 38 digits; DOUBLE precision defaults to 308 digits. DTS uses ROUND(COLUMN,PRECISION) to retrieve these values. Verify that precision settings meet your business requirements before starting the migration.

• If the source contains TTL indexes, data inconsistency may occur after migration.

Azure Cosmos DB and Amazon DocumentDB limitations

• For Azure Cosmos DB for MongoDB clusters and Amazon DocumentDB elastic clusters: only full data migration is supported.

• For inelastic Amazon DocumentDB clusters: enable change streams and set Migration Method to ChangeStream and Architecture to Sharded Cluster.

Other operational notes

• Evaluate the impact of migration on source and destination database performance before starting. Run the migration during off-peak hours when possible. Full data migration uses read and write resources on both databases and may increase server load.

• If a DTS task fails, DTS technical support attempts to restore it within 8 hours. During restoration, the task may restart and parameters may be modified. Only DTS task parameters are modified — database parameters are not changed. Parameters that may be modified include those described in Modify the parameters of a DTS instance.