Data Transmission Service:Notes and limits for migrating data from a MongoDB source database

Type

Description

Source database limits

• Bandwidth requirements: The server of the source database must have sufficient egress bandwidth. Insufficient bandwidth affects the migration speed.

• The collections to be migrated must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may exist in the destination database.

• If you migrate objects at the collection level and need to edit them, such as mapping collection names, a single data migration task can migrate a maximum of 1,000 collections. If you exceed this limit, an error is reported after you submit the task. In this case, split the collections into multiple data migration tasks or configure a task to migrate the entire database.

• A single piece of data to be migrated from the source database cannot exceed 16 MB. Otherwise, the task fails.

• Source database operation limits:

  • During schema migration and full data migration, do not change the schema of databases or collections. This includes updating data in arrays. Schema changes can cause the data migration task to fail or lead to data inconsistency between the source and destination databases.

  • This migration task does not support incremental data migration. To ensure data consistency, do not write new data to the source MongoDB database during full data migration.

• If a collection to be migrated contains a Time To Live (TTL) index, data inconsistency or instance latency may occur.

Other limits

• If the destination instance is a sharded cluster instance:

  • You must purge orphaned documents. Otherwise, migration performance is affected. If documents with conflicting _id values are found during migration, data inconsistency or task failure may occur.

  • Before the task starts, add the corresponding sharding keys of the destination instance to the source data that you want to migrate. If you cannot add sharding keys to the source instance, see Migrate data from a MongoDB instance without a sharding key to a MongoDB sharded cluster instance.

  • After the task starts, the data to be migrated must include a sharding key when you use the INSERT command. You cannot change the sharding key when you use the UPDATE command.

• If the destination instance is a ReplicaSet instance:

  • When the Access Method is Express Connect, VPN Gateway, or Smart Access Gateway, Public IP Address, or Cloud Enterprise Network (CEN), you must enter the address and port of the primary node or configure a high-availability (HA) connection address in the Domain Name or IP and Port Number fields. For more information about high-availability (HA) connection addresses, see Create an instance for a high-availability MongoDB source or destination database.

  • If Access Method is set to Self-managed Database on ECS, enter the port of the primary node for Port Number.

• Incremental migration is not supported.

• You cannot connect to a MongoDB database using an SRV record.

• If the destination collection has a unique index or its capped property is set to true, the collection does not support concurrent replay during incremental migration. It only supports single-threaded writes. This may increase task latency.

• Data in the admin, config, and local databases cannot be migrated.

• Transaction information is not retained. Transactions in the source database are converted into individual records when migrated to the destination database.

• If a primary key or unique key conflict occurs when DTS writes data to a destination collection, DTS skips the corresponding data write statement and retains the existing data in the destination collection.

• We recommend that you keep the database versions of the source and destination MongoDB instances the same, or migrate data from an earlier version to a later version to ensure compatibility. If you migrate data from a later version to an earlier version, database compatibility issues may occur.

• If the source MongoDB instance is earlier than V3.6 and the destination MongoDB instance is V3.6 or later, the order of fields in the data may be inconsistent after migration. This is because of differences in the execution plans of the database engines. The mapping between fields and values remains consistent. If your business involves match query logic for nested structures, evaluate the potential impact of the inconsistent field order.

• Before data migration, evaluate the performance of the source and destination databases. We recommend that you perform data migration during off-peak hours. During full data migration, DTS consumes some read and write resources of the source and destination databases. This may increase the database load.

• Full data migration involves concurrent INSERT operations, which cause fragmentation in the destination collections. After full data migration is complete, the disk space used by the destination collections is larger than that used by the source collections.

• Confirm whether the migration precision for columns of the FLOAT or DOUBLE data type meets your business requirements. DTS reads the values of these columns using ROUND(COLUMN,PRECISION). If the precision is not explicitly defined, DTS migrates FLOAT values with a precision of 38 digits and DOUBLE values with a precision of 308 digits.

• DTS attempts to resume a failed migration task within seven days. Before you switch your business to the destination instance, end or release the task. You can also use the revoke command to revoke the write permissions of the DTS account on the destination instance. This prevents the source data from overwriting the data in the destination instance after the task is automatically resumed.

• DTS writes data concurrently. This causes the storage space used by the destination instance to be 5% to 10% larger than that of the source instance.

• To query the count of documents in the destination MongoDB instance, use the db.$table_name.aggregate([{ $count:"myCount"}]) syntax.

• Ensure that the destination MongoDB instance does not have the same primary key (default is _id) as the source instance. Otherwise, data loss may occur. If the destination instance has the same primary key as the source instance, clear the relevant data in the destination instance without affecting your business. This means deleting documents in the destination instance that have the same _id as the source instance.

• If an instance fails, DTS helpdesk will try to recover the instance within 8 hours. During the recovery process, operations such as restarting the instance or adjusting its parameters may be performed.

  Note

  When parameters are adjusted, only the parameters of the DTS instance are modified. The parameters in the database are not modified. The parameters that may be modified include but are not limited to those described in Modify instance parameters.

• If the destination database is a MongoDB sharded cluster instance, ensure that your business operations comply with the requirements for sharded collections after you switch your business to this database.

• If the source database is MongoDB 5.0 or later and the destination database is earlier than 5.0, you cannot migrate a capped collection. This can cause the task to fail or lead to data inconsistency between the source and destination databases. This is because the behavior of a capped collection changed in MongoDB 5.0. It now allows explicit deletion and increases in document size during updates. Earlier database kernels are not compatible with these new features.

• Time-series collections introduced in MongoDB 5.0 and later cannot be migrated.

Special cases

If the source database is a self-managed MongoDB instance, a primary/secondary switchover during migration will cause the migration task to fail.

Type

Description

Source database limits

• Bandwidth requirements: The server of the source database must have sufficient egress bandwidth. Insufficient bandwidth affects the migration speed.

• The collections to be migrated must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may exist in the destination database.

• If you migrate objects at the collection level and need to edit them, such as mapping collection names, a single data migration task can migrate a maximum of 1,000 collections. If you exceed this limit, an error is reported after you submit the task. In this case, split the collections into multiple data migration tasks or configure a task to migrate the entire database.

• A single piece of data to be migrated from the source database cannot exceed 16 MB. Otherwise, the task fails.

• If the source database is Azure Cosmos DB for MongoDB or Amazon DocumentDB (with MongoDB compatibility) elastic cluster, only full migration is supported.

• For incremental migration:

  The source database must have Oplog enabled, and the Oplog must be retained for at least seven days. Alternatively, enable Change Streams and ensure that DTS can subscribe to data changes from the source database within the last seven days through Change Streams. Otherwise, the task may fail because it cannot get data changes from the source database. In extreme cases, this may lead to data inconsistency or data loss. Issues caused by this are not covered by the DTS Service-Level Agreement (SLA).

  Important

  • We recommend that you get data changes from the source database through Oplog.

  • Only MongoDB 4.0 and later support getting data changes through Change Streams.

  • If the source database is Amazon DocumentDB (a non-elastic cluster), you must manually enable Change Streams. When you configure the task, set Migration Method to ChangeStream and Architecture to Sharded Cluster.

• Source database operation limits:

  • During schema migration and full data migration, do not change the schema of databases or collections. This includes updating data in arrays. Schema changes can cause the data migration task to fail or lead to data inconsistency between the source and destination databases.

  • If you perform only full data migration, do not write new data to the source instance. Otherwise, data inconsistency between the source and destination databases occurs. To maintain real-time data consistency, select schema migration, full data migration, and incremental data migration.

• If a collection to be migrated contains a Time To Live (TTL) index, data inconsistency or instance latency may occur.

Other limits

• If the destination instance is a sharded cluster instance:

  • You must purge orphaned documents. Otherwise, migration performance is affected. If documents with conflicting _id values are found during migration, data inconsistency or task failure may occur.

  • Before the task starts, add the corresponding sharding keys of the destination instance to the source data that you want to migrate. If you cannot add sharding keys to the source instance, see Migrate data from a MongoDB instance without a sharding key to a MongoDB sharded cluster instance.

  • After the task starts, the data to be migrated must include a sharding key when you use the INSERT command. You cannot change the sharding key when you use the UPDATE command.

• If the destination instance is a ReplicaSet instance:

  • When the Access Method is Express Connect, VPN Gateway, or Smart Access Gateway, Public IP Address, or Cloud Enterprise Network (CEN), you must enter the address and port of the primary node or configure a high-availability (HA) connection address in the Domain Name or IP and Port Number fields. For more information about high-availability (HA) connection addresses, see Create an instance for a high-availability MongoDB source or destination database.

  • If Access Method is set to Self-managed Database on ECS, enter the port of the primary node for Port Number.

• You cannot connect to a MongoDB database using an SRV record.

• We recommend that you keep the database versions of the source and destination MongoDB instances the same, or migrate data from an earlier version to a later version to ensure compatibility. If you migrate data from a later version to an earlier version, database compatibility issues may occur.

• Data in the admin, config, and local databases cannot be migrated.

• If the destination collection has a unique index or its capped property is set to true, the collection does not support concurrent replay during incremental migration. It only supports single-threaded writes. This may increase task latency.

• Transaction information is not retained. Transactions in the source database are converted into individual records when migrated to the destination database.

• If a primary key or unique key conflict occurs when DTS writes data to a destination collection, DTS skips the corresponding data write statement and retains the existing data in the destination collection.

• If the source MongoDB instance is earlier than V3.6 and the destination MongoDB instance is V3.6 or later, the order of fields in the data may be inconsistent after migration. This is because of differences in the execution plans of the database engines. The mapping between fields and values remains consistent. If your business involves match query logic for nested structures, evaluate the potential impact of the inconsistent field order.

• Before data migration, evaluate the performance of the source and destination databases. We recommend that you perform data migration during off-peak hours. During full data migration, DTS consumes some read and write resources of the source and destination databases. This may increase the database load.

• Full data migration involves concurrent INSERT operations, which cause fragmentation in the destination collections. After full data migration is complete, the disk space used by the destination collections is larger than that used by the source collections.

• Confirm whether the migration precision for columns of the FLOAT or DOUBLE data type meets your business requirements. DTS reads the values of these columns using ROUND(COLUMN,PRECISION). If the precision is not explicitly defined, DTS migrates FLOAT values with a precision of 38 digits and DOUBLE values with a precision of 308 digits.

• DTS attempts to resume a failed migration task within seven days. Before you switch your business to the destination instance, end or release the task. You can also use the revoke command to revoke the write permissions of the DTS account on the destination instance. This prevents the source data from overwriting the data in the destination instance after the task is automatically resumed.

• DTS writes data concurrently. This causes the storage space used by the destination instance to be 5% to 10% larger than that of the source instance.

• To query the count of documents in the destination MongoDB instance, use the db.$table_name.aggregate([{ $count:"myCount"}]) syntax.

• Ensure that the destination MongoDB instance does not have the same primary key (default is _id) as the source instance. Otherwise, data loss may occur. If the destination instance has the same primary key as the source instance, clear the relevant data in the destination instance without affecting your business. This means deleting documents in the destination instance that have the same _id as the source instance.

• If an instance fails, DTS helpdesk will try to recover the instance within 8 hours. During the recovery process, operations such as restarting the instance or adjusting its parameters may be performed.

  Note

  When parameters are adjusted, only the parameters of the DTS instance are modified. The parameters in the database are not modified. The parameters that may be modified include but are not limited to those described in Modify instance parameters.

• If the destination database is a MongoDB sharded cluster instance, ensure that your business operations comply with the requirements for sharded collections after you switch your business to this database.

• If the source database is MongoDB 5.0 or later and the destination database is earlier than 5.0, you cannot migrate a capped collection. This can cause the task to fail or lead to data inconsistency between the source and destination databases. This is because the behavior of a capped collection changed in MongoDB 5.0. It now allows explicit deletion and increases in document size during updates. Earlier database kernels are not compatible with these new features.

• Time-series collections introduced in MongoDB 5.0 and later cannot be migrated.

Special cases

If the source database is a self-managed MongoDB instance:

• A primary/secondary switchover during migration will cause the migration task to fail.

• The latency of DTS is calculated by comparing the timestamp of the last migrated data record with the current timestamp. If the source database has not been updated for a long time, the latency information may be inaccurate. If the task shows a long latency, you can perform an update operation in the source database to refresh the latency information.

Type

Description

Source database limits

• Bandwidth requirements: The server of the source database must have sufficient egress bandwidth. Insufficient bandwidth affects the migration speed.

• The collections to be migrated must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may exist in the destination database.

• The _id field in the collections to be migrated must be unique. Otherwise, data inconsistency may occur.

• If you migrate objects at the collection level and need to edit them, such as mapping collection names, a single data migration task can migrate a maximum of 1,000 collections. If you exceed this limit, an error is reported after you submit the task. In this case, split the collections into multiple data migration tasks or configure a task to migrate the entire database.

• A single piece of data to be migrated from the source database cannot exceed 16 MB. Otherwise, the task fails.

• During full data migration, DTS consumes some resources of the source and destination databases, which may increase the database server load. If the database has a heavy workload or the server specifications are low, the database pressure may increase, and the database service may even become unavailable. Evaluate this carefully before performing data migration. We recommend that you perform data migration during off-peak hours.

• For information about the supported versions and storage engines for MongoDB instances, see Versions and storage engines. To migrate across versions or storage engines, confirm compatibility in advance.

• If the source database is Azure Cosmos DB for MongoDB or Amazon DocumentDB (with MongoDB compatibility) elastic cluster, only full migration is supported.

• For incremental migration:

  The source database must have Oplog enabled, and the Oplog must be retained for at least seven days. Alternatively, enable Change Streams and ensure that DTS can subscribe to data changes from the source database within the last seven days through Change Streams. Otherwise, the task may fail because it cannot get data changes from the source database. In extreme cases, this may lead to data inconsistency or data loss. Issues caused by this are not covered by the DTS Service-Level Agreement (SLA).

  Important

  • We recommend that you get data changes from the source database through Oplog.

  • Only MongoDB 4.0 and later support getting data changes through Change Streams.

  • If the source database is Amazon DocumentDB (a non-elastic cluster), you must manually enable Change Streams. When you configure the task, set Migration Method to ChangeStream and Architecture to Sharded Cluster.

• If you migrate objects at the collection level and need to edit them, such as mapping collection names, a single data migration task can migrate a maximum of 1,000 collections. If you exceed this limit, an error is reported after you submit the task. In this case, split the collections into multiple data migration tasks or configure a task to migrate the entire database.

• Data in the admin, config, and local databases cannot be migrated.

• If your MongoDB instance is version 8.0 or later and the Migration Method is set to Oplog, you must ensure that the shard account used for the migration task has the directShardOperations permission. You can run the db.adminCommand({ grantRolesToUser: "username", roles: [{ role: "directShardOperations", db: "admin"}]}) command to grant the permission.

  Note

  In the command, replace username with the shard account used by the migration task.

• The number of Mongos nodes in the source self-managed MongoDB sharded cluster instance cannot exceed 10.

• If a collection to be migrated contains a Time To Live (TTL) index, data inconsistency or instance latency may occur.

• Ensure that there are no orphaned documents in the source and destination MongoDB sharded cluster instances. Otherwise, data inconsistency or even task failure may occur. For more information, see Orphaned Document and How to clear orphaned documents in a MongoDB sharded cluster instance.

• Source database operation limits:

  • While the migration instance is running, do not run commands that change the data distribution of the objects to be migrated in the source database. Examples include shardCollection, reshardCollection, unshardCollection, moveCollection, and movePrimary. Otherwise, data inconsistency may occur.

  • During schema migration and full data migration, do not change the schema of databases or collections. This includes updating data in arrays. Schema changes can cause the data migration task to fail or lead to data inconsistency between the source and destination databases.

  • If you perform only full data migration, do not write new data to the source instance. Otherwise, data inconsistency between the source and destination databases occurs. To maintain real-time data consistency, select schema migration, full data migration, and incremental data migration.

• If the balancer in the source database is balancing data, instance latency may occur.

Other limits

• If you purchase a DTS task before configuring it, you must correctly enter the number of shards when you purchase the task.

• If the destination instance is a ReplicaSet instance:

  • When the Access Method is Express Connect, VPN Gateway, or Smart Access Gateway, Public IP Address, or Cloud Enterprise Network (CEN), you must enter the address and port of the primary node or configure a high-availability (HA) connection address in the Domain Name or IP and Port Number fields. For more information about high-availability (HA) connection addresses, see Create an instance for a high-availability MongoDB source or destination database.

  • If Access Method is set to Self-managed Database on ECS, enter the port of the primary node for Port Number.

• You cannot connect to a MongoDB database using an SRV record.

• If the target MongoDB is a sharded cluster instance and you do not need to use the schema migration feature provided by DTS (for example, if data sharding is already configured at the destination), do not select the Schema Migration checkbox under Migration Types on the Configure Objects page. Otherwise, sharding conflicts may cause data inconsistency or task failures.

• Before the task starts, add the corresponding sharding keys of the destination instance to the source data that you want to migrate. After the task starts, the data to be migrated must include a sharding key when you use the INSERT command. You cannot change the sharding key when you use the UPDATE command.

• During a full migration, you must disable the balancer for the source MongoDB database until each subtask enters the incremental phase. Otherwise, data inconsistency may occur. For more information, see Manage the MongoDB Balancer.

• Ensure that the destination MongoDB instance does not have the same primary key (default is _id) as the source instance. Otherwise, data loss may occur. If the destination instance has the same primary key as the source instance, clear the relevant data in the destination instance without affecting your business. This means deleting documents in the destination instance that have the same _id as the source instance.

• Transaction information is not retained. Transactions in the source database are converted into individual records when migrated to the destination database.

• If a primary key or unique key conflict occurs when DTS writes data to a destination collection, DTS skips the corresponding data write statement and retains the existing data in the destination collection.

• During a DTS migration, if you use the Oplog method for incremental migration, scaling the shards of a source MongoDB database with a sharded cluster architecture in or out is not supported. Otherwise, the DTS task will fail and data inconsistency will occur.

• To query the count of documents in the destination MongoDB instance, use the db.$table_name.aggregate([{ $count:"myCount"}]) syntax.

• DTS writes data concurrently. This causes the storage space used by the destination instance to be 5% to 10% larger than that of the source instance.

• If the destination collection has a unique index or its capped property is set to true, the collection does not support concurrent replay during incremental migration. It only supports single-threaded writes. This may increase task latency.

• If the source MongoDB instance is earlier than V3.6 and the destination MongoDB instance is V3.6 or later, the order of fields in the data may be inconsistent after migration. This is because of differences in the execution plans of the database engines. The mapping between fields and values remains consistent. If your business involves match query logic for nested structures, evaluate the potential impact of the inconsistent field order.

• Before data migration, evaluate the performance of the source and destination databases. We recommend that you perform data migration during off-peak hours. During full data migration, DTS consumes some read and write resources of the source and destination databases. This may increase the database load.

• Full data migration involves concurrent INSERT operations, which cause fragmentation in the destination tables. After full data migration is complete, the disk space used by the destination tables is larger than that used by the source tables.

• DTS attempts to resume a failed migration task within seven days. Before you switch your business to the destination instance, end or release the task. You can also use the revoke command to revoke the write permissions of the DTS account on the destination instance. This prevents the source data from overwriting the data in the destination instance after the task is automatically resumed.

• If an instance fails, DTS helpdesk will try to recover the instance within 8 hours. During the recovery process, operations such as restarting the instance or adjusting its parameters may be performed.

  Note

  When parameters are adjusted, only the parameters of the DTS instance are modified. The parameters in the database are not modified. The parameters that may be modified include but are not limited to those described in Modify instance parameters.

• If the destination database is a MongoDB sharded cluster instance, ensure that your business operations comply with the requirements for sharded collections after you switch your business to this database.

• If the source database is MongoDB 5.0 or later and the destination database is earlier than 5.0, you cannot migrate a capped collection. This can cause the task to fail or lead to data inconsistency between the source and destination databases. This is because the behavior of a capped collection changed in MongoDB 5.0. It now allows explicit deletion and increases in document size during updates. Earlier database kernels are not compatible with these new features.

• Time-series collections introduced in MongoDB 5.0 and later cannot be migrated.