ApsaraDB for MongoDB:Oplog best practices and risks

Oplog size

In ApsaraDB for MongoDB, the default oplog size is 10% of total disk space. For example, a 500 GB disk yields a 50 GB oplog. The oplog scales automatically when you increase disk space.

To adjust oplog size, modify the replication.oplogSizeMB parameter in the console. The change takes effect immediately without a restart. Set database parameters.

To check the oplog table size:

• Check the Disk Space Usage metric on the console monitoring page. Node Monitoring (formerly Basic Monitoring).

• Connect with mongo shell or mongosh and run the following command.

  rs.printReplicationInfo()

  Sample output:

  configured oplog size:   192MB
  log length start to end: 65422secs (18.17hrs)
  oplog first event time:  Mon Jun 23 2014 17:47:18 GMT-0400 (EDT)
  oplog last event time:   Tue Jun 24 2014 11:57:40 GMT-0400 (EDT)
  now:                     Thu Jun 26 2014 14:24:39 GMT-0400 (EDT)

  This shows a 192 MB oplog with an 18-hour oplog window.

Minimum oplog retention period

MongoDB 4.4 introduced the storage.oplogMinRetentionHours parameter to control the minimum oplog retention period and ensure a sufficient oplog window.

The default value is 0 (no minimum retention; cleanup is controlled by oplog size). When set, oplog cleanup occurs only when both conditions are met:

• The oplog usage exceeds the configured oplogSizeMB.

• The timestamps of the entries are older than the minimum oplog retention period.

Before the oplog reaches oplogSizeMB (for example, on a new instance), the actual oplog window can exceed the configured minimum retention period, and size is limited only by oplogSizeMB. After reaching oplogSizeMB, the retention period governs cleanup. High write rates can cause the oplog to grow well beyond oplogSizeMB.

To adjust the retention period, modify storage.oplogMinRetentionHours in the console. The change takes effect immediately without a restart. Set database parameters.

To view the retention period, check the Oplog retention duration metric on the console monitoring page. Node Monitoring (formerly Basic Monitoring).

ApsaraDB for MongoDB log backup

Log backup for ApsaraDB for MongoDB instances uses the oplog. A dedicated process continuously fetches the latest oplog entries and streams them to Object Storage Service (OSS) as log backup files. During point-in-time restores, these files replay the oplog.

In some cases, holes can appear in log backups, preventing point-in-time restores. Risks.

Configure oplog size or retention

The default oplog size is sufficient for most workloads. Consider increasing it for these scenarios:

• Frequent batch updates

  Each batch update generates multiple individual operations, producing many oplog entries.

• Frequent insertions and deletions

  Disk usage stays stable, but the oplog accumulates entries for both insert and delete operations.

• Numerous in-place updates to the same documents

  Updates that don't increase document size generate many oplog entries without noticeably changing disk usage.

Consider reducing oplog size for:

• Read-heavy and write-light workloads.

• Storing cold data.

Keep the oplog window at 24 hours or longer. For initial sync scenarios, the window must cover the time needed to copy all data, which depends on data volume, collection count, and instance specifications. In such cases, a longer oplog window may be necessary.

Monitor secondary node lag and configure alerts

If replication lag exceeds the oplog window, the secondary node enters an unrecoverable error state. Monitor replication lag and promptly submit a ticket if lag continuously increases.

Common causes:

• Network latency, packet loss, or interruptions.

• The secondary node's disk throughput is a bottleneck.

• A heavy write workload combined with a write concern of {w:1}.

• Kernel defects that block primary-secondary replication on the secondary node.

• Other unlisted reasons.

To check replication lag:

• View the Primary/Secondary Replication Latency metric on the console monitoring page. Node Monitoring (formerly Basic Monitoring).

• Connect with mongo shell or mongosh and run:

  rs.printSecondaryReplicationInfo()

  Sample output:

  source: m1.example.net:27017
      syncedTo: Thu Apr 10 2014 10:27:47 GMT-0400 (EDT)
      0 secs (0 hrs) behind the primary
  source: m2.example.net:27017
      syncedTo: Thu Apr 10 2014 10:27:47 GMT-0400 (EDT)
      0 secs (0 hrs) behind the primary

  Both secondary nodes show zero replication lag.

Create a CloudMonitor alert for the Replication Latency metric using the Alarm Rules feature. Set the threshold to 10 seconds or more. Set a threshold-based alert rule.

MongoDB versions earlier than 3.4

MongoDB 3.4 introduced periodic no-op writes to support the maxStalenessSeconds read preference parameter (SERVER-23892). These no-op writes advance the oplog during idle periods, enabling accurate lag measurement for secondary nodes.

On versions earlier than 3.4, the oplog stops advancing during idle periods. The log backup process cannot fetch new data, creating a backup hole that prevents point-in-time restores.

High write rate and short oplog window

Data from ApsaraDB for MongoDB instances shows that when the oplog generation rate reaches approximately 125 GB/h to 165 GB/h, the log backup process is highly likely to fall behind, creating a backup hole.

Estimate the generation rate by dividing oplog size by oplog window. For example, a 20 GB oplog with a 0.06 h window yields approximately 333.3 GB/h.

Typical scenarios:

• Data synchronization with DTS, mongoShake, or similar tools.

• Large batches of INSERT or UPDATE operations in a short period.

• Data seeding (rapid bulk import).

• Stress testing.

To prevent backup holes from high write rates:

• Apply rate limiting to synchronization tools by adjusting concurrency or batch size.

• Use a write concern of {w:"majority"} instead of {w:1}.

For workloads with inherently high oplog generation rates:

• Use a sharded cluster or add shards to distribute the oplog generation rate.

• Increase oplog size or minimum retention period to give the backup process more buffer to catch up during lower-traffic periods.