MongoShake, an open-source tool developed by Alibaba Cloud, synchronizes data between MongoDB databases for use cases such as data analytics, disaster recovery, and active-active replication. This topic explains how to configure real-time synchronization between ApsaraDB for MongoDB instances.
MongoShake
MongoShake is a general-purpose service tool developed by Alibaba Cloud in Go. It replicates data from MongoDB by reading its operation logs (oplogs) and replaying them on the destination instance.
MongoShake also provides data subscription and consumption features. It integrates with SDKs, Kafka, and MetaQ to support scenarios such as log subscription, data center synchronization, and asynchronous cache eviction.
For more information, see the MongoShake GitHub repository.
Supported data sources
Source | Destination |
Self-managed MongoDB database on an ECS instance | Self-managed MongoDB database on an ECS instance |
On-premises self-managed MongoDB database | On-premises self-managed MongoDB database |
ApsaraDB for MongoDB instance | ApsaraDB for MongoDB instance |
Third-party cloud MongoDB database | Third-party cloud MongoDB database |
Notes
Do not perform any DDL operation on the source database before the full sync is complete. Otherwise, data inconsistency may occur.
Synchronization of the local database is not supported. To synchronize data from the admin database, see Migrate business data from the admin database to a non-admin database.
Required permissions
Source | Required permissions |
Source MongoDB instance | The Note The |
Destination MongoDB instance | The |
For information about how to create and authorize MongoDB database users, see Use DMS to manage MongoDB database users or db.createUser command.
Prerequisites
For optimal synchronization performance, ensure that the source MongoDB replica set instance uses a VPC. If it uses the classic network, switch its network type to VPC. For more information, see Switch the network type of an instance from classic network to VPC.
Create a destination MongoDB replica set instance. To minimize network latency, create the destination instance in the same VPC as the source instance. For more information, see Create a replica set instance.
Create an ECS instance to run MongoShake. To minimize network latency, create the ECS instance in the same VPC as the source instance. For more information, see Create an ECS instance.
Add the private IP address of the ECS instance to the whitelists of both the source and destination MongoDB instances, and verify that the ECS instance can connect to both instances. For more information, see Modify a whitelist.
If your network setup does not meet the preceding requirements, you can apply for a public endpoint for both the source and destination MongoDB instances. Then, add the public IP address of the ECS instance to the whitelists of both MongoDB instances to perform synchronization over the public endpoints. For more information, see Apply for a public endpoint and Modify a whitelist.
Procedure
This procedure assumes that MongoShake is installed in the /test/mongoshake directory.
Log on to the ECS instance.
NoteYou can choose a logon method based on your scenario. For more information, see Overview of logon methods for ECS servers.
Run the following command to download the MongoShake program and rename it to
mongoshake.tar.gz.wget "https://github.com/alibaba/MongoShake/releases/download/release-v2.8.7-20251218/mongo-shake-v2.8.7.tgz" -O mongoshake.tar.gzNoteThe command downloads MongoShake 2.8.7. To download the latest version, see the releases page.
Run the following command to extract the MongoShake package to the /test/mongoshake directory.
tar zxvf mongoshake.tar.gz && mv mongo-shake-v2.8.7 /test/mongoshake && cd /test/mongoshakeRun the
vi collector.confcommand to modify the MongoShake configuration file collector.conf. The following table describes the main parameters.Parameter
Description
Example
mongo_urls
The connection string URI of the source MongoDB instance. In the example, the database account is test, which is in the admin database.
NoteWe recommend using a VPC endpoint to minimize network latency.
For details about the connection string URI format, see Connection description for a replica set instance.
mongo_urls = mongodb://test:****@dds-bp19f409d7512****.mongodb.rds.aliyuncs.com:3717,dds-bp19f409d7512****.mongodb.rds.aliyuncs.com:3717NoteThe password cannot contain the at sign (
@). Otherwise, the connection fails.tunnel.address
The connection string URI of the destination MongoDB instance. In the example, the database account is test, which is in the admin database.
NoteWe recommend using a VPC endpoint to minimize network latency.
For details about the connection string URI format, see Connection description for a replica set instance.
tunnel.address = mongodb://test:****@dds-bp19f409d7512****.mongodb.rds.aliyuncs.com:3717,dds-bp19f409d7512****.mongodb.rds.aliyuncs.com:3717NoteThe password cannot contain the at sign (
@). Otherwise, the connection fails.sync_mode
The data synchronization mode. Valid values:
all: performs a full sync and an incremental sync.
full: performs only a full sync.
incr: performs only an incremental sync.
NoteThe default value is
incr.sync_mode = allNoteFor a complete list of parameters in collector.conf, see the "collector.conf parameters" section in Appendix.
Run the following command to start the synchronization task and print logs.
./collector.linux -conf=collector.conf -verboseObserve the log output. When an entry similar to the following appears, it indicates that the full sync is complete and the incremental sync has started.
[09:38:57 CST 2019/06/20] [INFO] (mongoshake/collector.(*ReplicationCoordinator).Run:80) finish full sync, start incr sync with timestamp: fullBeginTs[1560994443], fullFinishTs[1560994737]
Monitoring status
After the incremental sync starts, open a new terminal window and run the following commands:
cd /test/mongoshake && ./mongoshake-stat --port=9100mongoshake-stat is a Python script and requires Python 2.7 to be installed. For more information, see the official Python website.
Sample output:
./mongoshake-stat --port=9100
logs_get/sec | logs_repl/sec | logs_success/sec | lsn.time | lsn_ack.time | lsn_ckpt.time | now.time | replset
none | none | none | 2019-06-20 10:34:14 | 2019-06-20 10:34:13 | 2019-06-20 10:33:36 | 2019-06-20 10:34:14 | mgset-10
545 | 768 | 768 | 2019-06-20 10:34:15 | 2019-06-20 10:34:15 | 2019-06-20 10:34:14 | 2019-06-20 10:34:15 | mgset-10
541 | 511 | 511 | 2019-06-20 10:34:15 | 2019-06-20 10:34:15 | 2019-06-20 10:34:14 | 2019-06-20 10:34:16 | mgset-10
639 | 512 | 512 | 2019-06-20 10:34:16 | 2019-06-20 10:34:16 | 2019-06-20 10:34:14 | 2019-06-20 10:34:17 | mgset-10
543 | 512 | 512 | 2019-06-20 10:34:17 | 2019-06-20 10:34:17 | 2019-06-20 10:34:14 | 2019-06-20 10:34:18 | mgset-10
546 | 768 | 768 | 2019-06-20 10:34:19 | 2019-06-20 10:34:19 | 2019-06-20 10:34:14 | 2019-06-20 10:34:19 | mgset-10
538 | 512 | 512 | 2019-06-20 10:34:20 | 2019-06-20 10:34:20 | 2019-06-20 10:34:19 | 2019-06-20 10:34:20 | mgset-10
550 | 511 | 511 | 2019-06-20 10:34:21 | 2019-06-20 10:34:21 | 2019-06-20 10:34:19 | 2019-06-20 10:34:21 | mgset-10
546 | 512 | 512 | 2019-06-20 10:34:22 | 2019-06-20 10:34:22 | 2019-06-20 10:34:19 | 2019-06-20 10:34:22 | mgset-10
531 | 512 | 512 | 2019-06-20 10:34:23 | 2019-06-20 10:34:23 | 2019-06-20 10:34:19 | 2019-06-20 10:34:23 | mgset-10
547 | 512 | 512 | 2019-06-20 10:34:24 | 2019-06-20 10:34:24 | 2019-06-20 10:34:19 | 2019-06-20 10:34:24 | mgset-10
541 | 512 | 512 | 2019-06-20 10:34:24 | 2019-06-20 10:34:24 | 2019-06-20 10:34:19 | 2019-06-20 10:34:25 | mgset-10Parameter descriptions:
Parameter | Description |
logs_get/sec | The number of oplogs fetched per second. |
logs_repl/sec | The number of oplog replay operations performed per second. |
logs_success/sec | The number of successful oplog replay operations per second. |
lsn.time | The timestamp of the last oplog sent. |
lsn_ack.time | The timestamp when the write operation was confirmed by the destination instance. |
lsn_ckpt.time | The timestamp when the checkpoint was persisted. |
now.time | The current time. |
replset | The name of the source database replica set. |
Migrate data from the admin database
MongoDB does not officially recommend storing business data in the admin database because locking behavior and conflicts with internal commands can degrade instance performance.
MongoShake supports synchronizing business data from the admin database to a non-admin database.
To do this, follow the Procedure. In Step 4, when you modify the collector.conf file, add the following configuration items:
filter.pass.special.db = admin
# Migrate all business collections from the admin database to newDB.
transform.namespace = admin:newDB
# Or, migrate the abc collection in the admin database to the def collection in the target database. You can configure multiple rules.
transform.namespace = admin.abc:target.defAppendix
Table 1. collector.conf parameters
Category | Parameter | Description | Example |
N/A | conf.version | The version of the configuration file. Do not modify this value. |
|
Global options | id | The ID of the sync task. You can customize this value. It is used for the log file name, the database name for storing checkpoint information, and the name of the database on the destination instance. |
|
master_quorum | The high availability option. When primary and standby MongoShake nodes synchronize data from the same source, set this parameter to Valid values:
Note The default value is false. |
| |
full_sync.http_port | The HTTP port for viewing the full sync status. Note The default value is 9101. |
| |
incr_sync.http_port | The HTTP port for viewing the incremental sync status. Note The default value is 9100. |
| |
system_profile_port | The profiling port that is used to view internal stack information. |
| |
log.level | The log level. Valid values:
Default value: info. |
| |
log.dir | The directory for log files and PID files. If this parameter is not set, the logs directory in the current path is used by default. Note You must specify an absolute path for this parameter. |
| |
log.file | The name of the log file. You can customize this value. Note The default value is collector.log. |
| |
log.flush | The log refresh frequency on the screen. Valid values:
Note The default value is false. |
| |
sync_mode | The data synchronization mode. Valid values:
Note The default value is incr. |
| |
mongo_urls | The connection string URI of the source MongoDB instance. In the example, the database account is test, which is in the admin database. Note
|
| |
mongo_cs_url | If the source is a sharded cluster instance, you must specify the connection string URI of the ConfigServer (CS) node. For information about how to obtain the connection string URI of a ConfigServer node, see Apply for an endpoint for a shard. In the example, the database account is test, which is in the admin database. |
| |
mongo_s_url | If the source is a sharded cluster instance, you must specify the connection string URI of at least one mongos node. Separate multiple mongos addresses with commas (,). For information about how to obtain the connection string URI of a mongos node, see Apply for an endpoint for a shard. In the example, the database account is test, which is in the admin database. |
| |
tunnel | The synchronization channel type. Valid values:
Note The default value is direct. |
| |
tunnel.address | The address of the destination. The supported address format varies based on the tunnel type:
In the example, the database account is test, which is in the admin database. |
| |
tunnel.message | The data type in the channel. This parameter is valid only when
Note The default value is raw. |
| |
mongo_connect_mode | The connection mode of the MongoDB instance. This parameter is valid only when
Note The default value is secondaryPreferred. |
| |
filter.namespace.black | Specifies a blacklist for data synchronization. The specified namespaces are not synchronized to the destination database. Separate multiple namespaces with semicolons (;). Note A namespace is the canonical name of a collection or index in MongoDB. It is a combination of the database name and the collection or index name, such as |
| |
filter.namespace.white | Specifies a whitelist for data synchronization. Only the specified namespaces are synchronized to the destination database. Separate multiple namespaces with semicolons (;). |
| |
filter.pass.special.db | Enables synchronization of special databases. By default, databases such as |
| |
filter.ddl_enable | Specifies whether to enable DDL synchronization. Valid values:
Note This feature is not supported when the source is a sharded cluster instance. |
| |
checkpoint.storage.url | The storage address for checkpoints, which is used to support resumable synchronization. If this parameter is not configured, the program writes checkpoints to one of the following databases based on the instance type:
In the example, the database account is test, which is in the admin database. |
| |
checkpoint.storage.db | The name of the database for storing checkpoints. Note The default value is mongoshake. |
| |
checkpoint.storage.collection | The name of the collection for storing checkpoints. When primary and standby MongoShake nodes synchronize data from the same source, you can change this collection name to prevent conflicts. Note The default value is ckpt_default. |
| |
checkpoint.start_position | The start position for the sync. This parameter is used only if no checkpoint is found. The format is Note The default value is 1970-01-01T00:00:00Z. |
| |
transform.namespace | Renames a source database or collection and synchronizes it to the destination. For example, you can rename the |
| |
Full sync options | full_sync.reader.collection_parallel | The maximum number of collections that MongoShake can pull concurrently at a time. |
|
full_sync.reader.write_document_parallel | The number of concurrent write threads per collection for MongoShake. |
| |
full_sync.reader.document_batch_size | The batch size for writing documents to the destination instance. For example, a value of 128 indicates that 128 documents are batched together before being written. |
| |
full_sync.collection_exist_drop | Specifies what to do if a destination collection has the same name as a source collection. Valid values:
|
| |
full_sync.create_index | Specifies whether to create an index after the synchronization is complete. Valid values:
|
| |
full_sync.executor.insert_on_dup_update | Specifies whether to convert an
|
| |
full_sync.executor.filter.orphan_document | Specifies whether to filter orphaned documents if the source is a sharded cluster instance. Valid values:
|
| |
full_sync.executor.majority_enable | Specifies whether to enable majority write on the destination instance. Valid values:
|
| |
Incremental sync options | incr_sync.mongo_fetch_method | The method for pulling incremental data. Valid values:
Default value: oplog. |
|
incr_sync.oplog.gids | Used to set up bidirectional replication for cloud clusters. |
| |
incr_sync.shard_key | The internal concurrency method for MongoShake. Do not modify this parameter. |
| |
incr_sync.worker | The number of concurrent threads for transmitting oplogs. You can increase this value if the host performance is sufficient. Note If the source is a sharded cluster instance, the number of threads must be equal to the number of shards. |
| |
incr_sync.worker.oplog_compressor | Enables data compression to reduce network bandwidth consumption. Valid values:
Note This parameter is valid only when the |
| |
incr_sync.target_delay | The delay, in seconds, before changes from the source are applied to the destination. You can set this parameter to prevent accidental operations. For example, Note A value of 0 indicates real-time synchronization. |
| |
incr_sync.worker.batch_queue_size | The internal queue configuration parameters of MongoShake. Do not modify these parameters unless necessary. |
| |
incr_sync.adaptive.batching_max_size |
| ||
incr_sync.fetcher.buffer_capacity |
| ||
MongoDB sync options (for | incr_sync.executor.upsert | Specifies whether to convert an
|
|
incr_sync.executor.insert_on_dup_update | Specifies whether to convert an
|
| |
incr_sync.conflict_write_to | Specifies whether to record conflicting documents if a write conflict occurs during synchronization. Valid values:
|
| |
incr_sync.executor.majority_enable | Specifies whether to enable majority write on the destination instance. Valid values:
Note Enabling this feature can affect performance. |
|
FAQ
Refer to the MongoShake FAQ. If you encounter other issues when you use MongoShake, you can file an issue on GitHub Issues.