Data Transmission Service:Notes and limits for migrating data from a MongoDB source

Type

Description

Source database limits

• Bandwidth requirements: The server that hosts the source database must have sufficient outbound bandwidth. Otherwise, the migration speed is affected.

• The collections to be migrated must have primary keys or UNIQUE constraints, and the fields must be unique. Otherwise, duplicate data may appear in the destination database.

• If you migrate data at the collection level and need to edit the collections, such as mapping collection names, a single data migration task can migrate a maximum of 1,000 collections. If you exceed this limit, an error is reported when you submit the task. In this case, split the collections into multiple migration tasks or configure a task to migrate the entire database.

• A single piece of data to be migrated from the source database cannot exceed 16 MB. Otherwise, the task fails.

• Source database operation limits:

  • During schema migration and full data migration, do not change the schema of databases or collections. This includes updating data of the array type. Otherwise, the data migration task may fail, or data inconsistency may occur between the source and destination databases.

  • This migration scenario does not support incremental data migration. To ensure data consistency, do not write new data to the source MongoDB database during full data migration.

• If a collection to be migrated contains a TTL (Time To Live) index, data inconsistency or instance latency may occur.

Other limits

• If the destination instance is a sharded cluster instance:

  • Purge orphaned documents. Otherwise, migration performance is affected. If documents with conflicting _id values are found during migration, data inconsistency or task failure may occur.

  • Before the task starts, add a sharding key to the source data that corresponds to the sharding key in the destination instance. If you cannot add a sharding key to the source data, see Migrate data from a MongoDB instance without a sharding key to a MongoDB sharded cluster instance.

  • After the task starts, the data to be migrated must include the sharding key when you use the INSERT command. You cannot change the sharding key when you use the UPDATE command.

• If the destination instance is a replica set instance:

  • When Access Method is set to Express Connect, VPN Gateway, or Smart Access Gateway, Public IP Address, or Cloud Enterprise Network (CEN), you must set Domain Name or IP and Port Number to the address and port of the primary node, or configure a high-availability connection address. For more information about high-availability connection addresses, see Create an instance with a high-availability MongoDB source or destination database.

  • When Access Method is Self-managed Database on ECS, enter the port of the primary node for Port Number.

• Incremental migration is not supported.

• Connecting to a MongoDB database using an SRV record is not supported.

• If the destination collection has a unique index or its capped property is set to true, concurrent replay is not supported for the collection during incremental migration. Only single-threaded writes are supported. This may increase task latency.

• Data in the admin, config, and local databases cannot be migrated.

• Transaction information is not retained. Transactions from the source database are converted into individual records in the destination database.

• When DTS writes data to the destination collection, if a primary key or unique key conflict occurs, DTS skips the corresponding write statement and retains the existing data in the destination collection.

• We recommend that the source and destination MongoDB databases have the same version, or that you migrate from a lower version to a higher version to ensure compatibility. If you migrate from a higher version to a lower version, compatibility issues may occur.

• If the source is a MongoDB instance earlier than version 3.6 and the destination is a MongoDB instance of version 3.6 or later, the order of fields in the data may be inconsistent after migration. This is due to differences in the execution plans of the database engines. The field-value pairs remain consistent. If your business logic involves text match queries on nested structures, assess the potential impact of this inconsistency.

• Before you migrate data, assess the performance of the source and destination databases. We recommend that you migrate data during off-peak hours. During full data migration, DTS consumes some read and write resources of the source and destination databases, which may increase the database workload.

• Full data migration involves concurrent INSERT operations, which can cause fragmentation in the destination collections. After full data migration is complete, the disk space used by the destination collections will be larger than that of the source collections.

• Confirm whether the migration precision that DTS provides for columns of the FLOAT or DOUBLE data type meets your business requirements. DTS reads the values of these columns using ROUND(COLUMN,PRECISION). If you do not specify the precision, DTS migrates FLOAT values with a precision of 38 digits and DOUBLE values with a precision of 308 digits.

• DTS attempts to resume failed migration tasks within seven days. Before you switch your business to the destination instance, make sure to end or release the task, or use the revoke command to revoke the write permissions of the account that DTS uses to access the destination instance. This prevents the source data from overwriting the data in the destination instance after the task is automatically resumed.

• Because DTS writes data concurrently, the storage space used by the destination instance is 5% to 10% larger than that of the source instance.

• To query the number of documents in the destination MongoDB instance, use the db.$table_name.aggregate([{ $count:"myCount"}]) syntax.

• Make sure that the destination MongoDB instance does not have the same primary keys (the _id field by default) as the source instance. Otherwise, data loss may occur. If the destination instance has the same primary keys, clear the relevant data from the destination instance without affecting your business. This means deleting the documents in the destination instance that have the same _id values as the source instance.

• If an instance fails, DTS helpdesk will try to recover the instance within 8 hours. During the recovery process, operations such as restarting the instance and adjusting parameters may be performed.

  Note

  When parameters are adjusted, only the parameters of the DTS instance are modified. The parameters of the database are not modified. The parameters that may be modified include but are not limited to those described in Modify instance parameters.

• If the destination database is a MongoDB sharded cluster, after you switch your business to this database, you must ensure that your business operations comply with the requirements for sharded collections in that MongoDB database.

• If the source database is MongoDB 5.0 or later and the destination database is earlier than 5.0, you cannot migrate a capped collection. This can cause the task to fail or lead to data inconsistency between the source and destination databases. This is because the behavior of capped collection changed in MongoDB 5.0, allowing explicit deletions and document size increases on update. Earlier database kernels do not support these new features.

• Migrating time-series collections introduced in MongoDB 5.0 and later is not supported.

Special cases

If the source is a self-managed MongoDB database, a primary/secondary switchover during migration will cause the migration task to fail.

Type

Description

Source database limits

• Bandwidth requirements: The server that hosts the source database must have sufficient outbound bandwidth. Otherwise, the migration speed is affected.

• The collections to be migrated must have primary keys or UNIQUE constraints, and the fields must be unique. Otherwise, duplicate data may appear in the destination database.

• If you migrate data at the collection level and need to edit the collections, such as mapping collection names, a single data migration task can migrate a maximum of 1,000 collections. If you exceed this limit, an error is reported when you submit the task. In this case, split the collections into multiple migration tasks or configure a task to migrate the entire database.

• A single piece of data to be migrated from the source database cannot exceed 16 MB. Otherwise, the task fails.

• If the source database is Azure Cosmos DB for MongoDB or an Amazon DocumentDB elastic cluster, only full data migration is supported.

• To perform incremental migration:

  The source database must have the oplog enabled, and the oplog must be retained for at least seven days. Alternatively, enable change streams and ensure that DTS can subscribe to data changes from the source database within the last seven days through change streams. Otherwise, the task may fail because it cannot obtain data changes from the source database. In extreme cases, data inconsistency or data loss may occur. Issues caused by this are not covered by the DTS Service-Level Agreement (SLA).

  Important

  • We recommend that you obtain data changes from the source database through the oplog.

  • Only MongoDB 4.0 and later support obtaining data changes through change streams.

  • If the source database is an Amazon DocumentDB (non-elastic cluster), you must manually enable Change Streams, and set Migration Method to ChangeStream and Architecture to Sharded Cluster when you configure the task.

• Source database operation limits:

  • During schema migration and full data migration, do not change the schema of databases or collections. This includes updating data of the array type. Otherwise, the data migration task may fail, or data inconsistency may occur between the source and destination databases.

  • If you perform only full data migration, do not write new data to the source instance. Otherwise, data inconsistency occurs between the source and destination databases. To maintain real-time data consistency, select Schema Migration, Full Data Migration, and Incremental Data Migration.

• If a collection to be migrated contains a TTL (Time To Live) index, data inconsistency or instance latency may occur.

Other limits

• If the destination instance is a sharded cluster instance:

  • Purge orphaned documents. Otherwise, migration performance is affected. If documents with conflicting _id values are found during migration, data inconsistency or task failure may occur.

  • Before the task starts, add a sharding key to the source data that corresponds to the sharding key in the destination instance. If you cannot add a sharding key to the source data, see Migrate data from a MongoDB instance without a sharding key to a MongoDB sharded cluster instance.

  • After the task starts, the data to be migrated must include the sharding key when you use the INSERT command. You cannot change the sharding key when you use the UPDATE command.

• If the destination instance is a replica set instance:

  • When Access Method is set to Express Connect, VPN Gateway, or Smart Access Gateway, Public IP Address, or Cloud Enterprise Network (CEN), you must set Domain Name or IP and Port Number to the address and port of the primary node, or configure a high-availability connection address. For more information about high-availability connection addresses, see Create an instance with a high-availability MongoDB source or destination database.

  • When Access Method is Self-managed Database on ECS, enter the port of the primary node for Port Number.

• Connecting to a MongoDB database using an SRV record is not supported.

• We recommend that the source and destination MongoDB databases have the same version, or that you migrate from a lower version to a higher version to ensure compatibility. If you migrate from a higher version to a lower version, compatibility issues may occur.

• Data in the admin, config, and local databases cannot be migrated.

• If the destination collection has a unique index or its capped property is set to true, concurrent replay is not supported for the collection during incremental migration. Only single-threaded writes are supported. This may increase task latency.

• Transaction information is not retained. Transactions from the source database are converted into individual records in the destination database.

• When DTS writes data to the destination collection, if a primary key or unique key conflict occurs, DTS skips the corresponding write statement and retains the existing data in the destination collection.

• If the source is a MongoDB instance earlier than version 3.6 and the destination is a MongoDB instance of version 3.6 or later, the order of fields in the data may be inconsistent after migration. This is due to differences in the execution plans of the database engines. The field-value pairs remain consistent. If your business logic involves text match queries on nested structures, assess the potential impact of this inconsistency.

• Before you migrate data, assess the performance of the source and destination databases. We recommend that you migrate data during off-peak hours. During full data migration, DTS consumes some read and write resources of the source and destination databases, which may increase the database workload.

• Full data migration involves concurrent INSERT operations, which can cause fragmentation in the destination collections. After full data migration is complete, the disk space used by the destination collections will be larger than that of the source collections.

• Confirm whether the migration precision that DTS provides for columns of the FLOAT or DOUBLE data type meets your business requirements. DTS reads the values of these columns using ROUND(COLUMN,PRECISION). If you do not specify the precision, DTS migrates FLOAT values with a precision of 38 digits and DOUBLE values with a precision of 308 digits.

• DTS attempts to resume failed migration tasks within seven days. Before you switch your business to the destination instance, make sure to end or release the task, or use the revoke command to revoke the write permissions of the account that DTS uses to access the destination instance. This prevents the source data from overwriting the data in the destination instance after the task is automatically resumed.

• Because DTS writes data concurrently, the storage space used by the destination instance is 5% to 10% larger than that of the source instance.

• To query the number of documents in the destination MongoDB instance, use the db.$table_name.aggregate([{ $count:"myCount"}]) syntax.

• Make sure that the destination MongoDB instance does not have the same primary keys (the _id field by default) as the source instance. Otherwise, data loss may occur. If the destination instance has the same primary keys, clear the relevant data from the destination instance without affecting your business. This means deleting the documents in the destination instance that have the same _id values as the source instance.

• If an instance fails, DTS helpdesk will try to recover the instance within 8 hours. During the recovery process, operations such as restarting the instance and adjusting parameters may be performed.

  Note

  When parameters are adjusted, only the parameters of the DTS instance are modified. The parameters of the database are not modified. The parameters that may be modified include but are not limited to those described in Modify instance parameters.

• If the destination database is a MongoDB sharded cluster, after you switch your business to this database, you must ensure that your business operations comply with the requirements for sharded collections in that MongoDB database.

• If the source database is MongoDB 5.0 or later and the destination database is earlier than 5.0, you cannot migrate a capped collection. This can cause the task to fail or lead to data inconsistency between the source and destination databases. This is because the behavior of capped collection changed in MongoDB 5.0, allowing explicit deletions and document size increases on update. Earlier database kernels do not support these new features.

• Migrating time-series collections introduced in MongoDB 5.0 and later is not supported.

Special cases

If the source is a self-managed MongoDB database:

• A primary/secondary switchover during migration will cause the migration task to fail.

• DTS calculates latency by comparing the timestamp of the last migrated data record with the current timestamp. If the source database has not been updated for a long time, the latency information may be inaccurate. If the task shows high latency, you can perform an update operation on the source database to refresh the latency information.

Type

Description

Source database limits

• Bandwidth requirements: The server that hosts the source database must have sufficient outbound bandwidth. Otherwise, the migration speed is affected.

• The collections to be migrated must have primary keys or UNIQUE constraints, and the fields must be unique. Otherwise, duplicate data may appear in the destination database.

• The _id field in the collections to be migrated must be unique. Otherwise, data inconsistency may occur.

• If you migrate data at the collection level and need to edit the collections, such as mapping collection names, a single data migration task can migrate a maximum of 1,000 collections. If you exceed this limit, an error is reported when you submit the task. In this case, split the collections into multiple migration tasks or configure a task to migrate the entire database.

• A single piece of data to be migrated from the source database cannot exceed 16 MB. Otherwise, the task fails.

• During full data migration, DTS consumes some resources of the source and destination databases, which may increase the database server load. If the database has a heavy workload or the server specifications are low, the database pressure may increase, and the database service may even become unavailable. Before you migrate data, carefully assess the impact. We recommend that you migrate data during off-peak hours.

• For information about the supported versions and storage engines for MongoDB instances, see Versions and storage engines. To migrate across versions or storage engines, confirm compatibility in advance.

• If the source database is Azure Cosmos DB for MongoDB or an Amazon DocumentDB elastic cluster, only full data migration is supported.

• To perform incremental migration:

  The source database must have the oplog enabled, and the oplog must be retained for at least seven days. Alternatively, enable change streams and ensure that DTS can subscribe to data changes from the source database within the last seven days through change streams. Otherwise, the task may fail because it cannot obtain data changes from the source database. In extreme cases, data inconsistency or data loss may occur. Issues caused by this are not covered by the DTS Service-Level Agreement (SLA).

  Important

  • We recommend that you obtain data changes from the source database through the oplog.

  • Only MongoDB 4.0 and later support obtaining data changes through change streams.

  • If the source database is an Amazon DocumentDB (non-elastic cluster), you must manually enable Change Streams, and set Migration Method to ChangeStream and Architecture to Sharded Cluster when you configure the task.

• If you migrate data at the collection level and need to edit the collections, such as mapping collection names, a single data migration task can migrate a maximum of 1,000 collections. If you exceed this limit, an error is reported when you submit the task. In this case, split the collections into multiple migration tasks or configure a task to migrate the entire database.

• Data in the admin, config, and local databases cannot be migrated.

• If the MongoDB version is 8.0 or later and the Migration Method is Oplog, you must ensure that the shard account used for the migration task has the directShardOperations permission. You can run the db.adminCommand({ grantRolesToUser: "username", roles: [{ role: "directShardOperations", db: "admin"}]}) command to grant the permission.

  Note

  In the command, replace username with the shard account used by the migration task.

• The number of Mongos nodes in the source self-managed MongoDB sharded cluster instance cannot exceed 10.

• If a collection to be migrated contains a TTL (Time To Live) index, data inconsistency or instance latency may occur.

• Make sure that the source and destination MongoDB sharded cluster instances do not contain orphaned documents. Otherwise, data inconsistency or even task failure may occur. For more information, see Orphaned Document and How do I purge orphaned documents from a MongoDB sharded cluster?.

• Source database operation limits:

  • While the migration instance is running, do not run commands that change the data distribution of the objects to be migrated in the source database. Examples include shardCollection, reshardCollection, unshardCollection, moveCollection, and movePrimary. Otherwise, data inconsistency may occur.

  • During schema migration and full data migration, do not change the schema of databases or collections. This includes updating data of the array type. Otherwise, the data migration task may fail, or data inconsistency may occur between the source and destination databases.

  • If you perform only full data migration, do not write new data to the source instance. Otherwise, data inconsistency occurs between the source and destination databases. To maintain real-time data consistency, select Schema Migration, Full Data Migration, and Incremental Data Migration.

• If the balancer in the source database is balancing data, instance latency may occur.

Other limits

• If you purchase a DTS task before configuring it, you must correctly specify the number of shards when you purchase the task.

• If the destination instance is a replica set instance:

  • When Access Method is set to Express Connect, VPN Gateway, or Smart Access Gateway, Public IP Address, or Cloud Enterprise Network (CEN), you must set Domain Name or IP and Port Number to the address and port of the primary node, or configure a high-availability connection address. For more information about high-availability connection addresses, see Create an instance with a high-availability MongoDB source or destination database.

  • When Access Method is Self-managed Database on ECS, enter the port of the primary node for Port Number.

• Connecting to a MongoDB database using an SRV record is not supported.

• If the destination MongoDB is a sharded cluster instance and you do not need to use the schema migration feature provided by DTS (for example, if data sharding is already configured on the destination instance), do not select Schema Migration under Migration Types on the Configure Objects page. Otherwise, data inconsistency or task failure may occur due to sharding conflicts.

• Before the task starts, add a sharding key to the source data that corresponds to the sharding key in the destination instance. After the task starts, the data to be migrated must include the sharding key when you use the INSERT command. You cannot change the sharding key when you use the UPDATE command.

• During full data migration, you must disable the balancer of the source MongoDB database until each subtask enters the incremental migration phase. Otherwise, data inconsistency may occur. For information about how to manage the balancer, see Manage the MongoDB balancer.

• Make sure that the destination MongoDB instance does not have the same primary keys (the _id field by default) as the source instance. Otherwise, data loss may occur. If the destination instance has the same primary keys, clear the relevant data from the destination instance without affecting your business. This means deleting the documents in the destination instance that have the same _id values as the source instance.

• Transaction information is not retained. Transactions from the source database are converted into individual records in the destination database.

• When DTS writes data to the destination collection, if a primary key or unique key conflict occurs, DTS skips the corresponding write statement and retains the existing data in the destination collection.

• During DTS migration, scaling the number of shards in or out for the MongoDB sharded cluster is not supported. Otherwise, the DTS task fails.

• To query the number of documents in the destination MongoDB instance, use the db.$table_name.aggregate([{ $count:"myCount"}]) syntax.

• Because DTS writes data concurrently, the storage space used by the destination instance is 5% to 10% larger than that of the source instance.

• If the destination collection has a unique index or its capped property is set to true, concurrent replay is not supported for the collection during incremental migration. Only single-threaded writes are supported. This may increase task latency.

• If the source is a MongoDB instance earlier than version 3.6 and the destination is a MongoDB instance of version 3.6 or later, the order of fields in the data may be inconsistent after migration. This is due to differences in the execution plans of the database engines. The field-value pairs remain consistent. If your business logic involves text match queries on nested structures, assess the potential impact of this inconsistency.

• Before you migrate data, assess the performance of the source and destination databases. We recommend that you migrate data during off-peak hours. During full data migration, DTS consumes some read and write resources of the source and destination databases, which may increase the database workload.

• Full data migration involves concurrent INSERT operations, which can cause fragmentation in the destination tables. After full data migration is complete, the storage space used by the destination tables will be larger than that of the source tables.

• DTS attempts to resume failed migration tasks within seven days. Before you switch your business to the destination instance, make sure to end or release the task, or use the revoke command to revoke the write permissions of the account that DTS uses to access the destination instance. This prevents the source data from overwriting the data in the destination instance after the task is automatically resumed.

• If an instance fails, DTS helpdesk will try to recover the instance within 8 hours. During the recovery process, operations such as restarting the instance and adjusting parameters may be performed.

  Note

  When parameters are adjusted, only the parameters of the DTS instance are modified. The parameters of the database are not modified. The parameters that may be modified include but are not limited to those described in Modify instance parameters.

• If the destination database is a MongoDB sharded cluster, after you switch your business to this database, you must ensure that your business operations comply with the requirements for sharded collections in that MongoDB database.

• If the source database is MongoDB 5.0 or later and the destination database is earlier than 5.0, you cannot migrate a capped collection. This can cause the task to fail or lead to data inconsistency between the source and destination databases. This is because the behavior of capped collection changed in MongoDB 5.0, allowing explicit deletions and document size increases on update. Earlier database kernels do not support these new features.

• Migrating time-series collections introduced in MongoDB 5.0 and later is not supported.