ApsaraDB for MongoDB:How MongoDB replica sets work

Primary election

A replica set is initialized using replSetInitiate or rs.initiate(). After initialization, members exchange heartbeats and elect a primary. The node that receives a majority of votes becomes primary; the others become secondaries.

Initialize a replica set

    config = {
        _id : "my_replica_set",
        members : [
             {_id : 0, host : "rs1.example.net:27017"},
             {_id : 1, host : "rs2.example.net:27017"},
             {_id : 2, host : "rs3.example.net:27017"},
       ]
    }
    rs.initiate(config)

Definition of "majority"

For N voting members, a majority equals N/2 + 1. If fewer than a majority are active, the replica set cannot elect a primary and becomes read-only.

Use an odd number of members. A three-node and a four-node replica set both tolerate only one failure, but four nodes provide more reliable data storage.

Special secondary nodes

By default, a secondary participates in elections, can become primary, and synchronizes data from the primary to maintain an identical dataset.

Secondaries can serve read requests to increase read capacity. MongoDB supports several specialized secondary types for different scenarios.

• Arbiter

  An arbiter only votes in elections. It cannot become primary and does not store data.

  In a two-node replica set, if either node goes down, no primary can be elected. Adding an arbiter enables elections to succeed even when one data-bearing node is unavailable.

  Because arbiters are lightweight (no data storage), they are ideal for replica sets with an even number of members.

• Priority0

  A priority 0 node cannot be elected as primary.

  For example, in a multi-datacenter deployment, set the priority of members in datacenter B to 0 to ensure the primary always stays in datacenter A.

  Note

  A majority of nodes must be in datacenter A. Otherwise, no primary can be elected during a network partition.

• Vote 0

  In MongoDB 3.0, a replica set supports up to 50 members, but only seven can vote. Non-voting members (Vote0) must have their `vote` property set to 0.

• Hidden

  A hidden node has a priority of 0 and is invisible to the driver.

  Hidden nodes are ideal for data backup or offline computing because they do not serve client requests.

• Delayed

  A delayed node is a hidden node whose data lags behind the primary by a configurable period, such as one hour.

  Delayed nodes enable point-in-time recovery if incorrect data is written to the primary.

Primary re-election

Beyond initialization, a primary re-election occurs in these scenarios:

• Replica set reconfiguration

  A re-election is triggered when a secondary detects the primary is down, or when the primary voluntarily steps down. The result depends on heartbeats, priority, and the latest oplog time.

  • Node priority

    Nodes vote for the highest-priority candidate. A priority-0 node never initiates an election. If the primary discovers a higher-priority secondary whose data lag is less than 10 seconds, it steps down to let that secondary take over.

  • Optime

    Only the node with the latest optime (the timestamp of the most recent oplog entry) can be elected as primary.

• Network partition

  A node can become primary only if it connects to a majority of voting nodes. If the primary loses connectivity to a majority, it steps down to secondary. During a network partition, multiple primaries might briefly coexist. Set the write concern to majority to ensure only one primary can complete writes successfully.

Data synchronization

Primary-to-secondary data synchronization uses an oplog. Each write on the primary creates an entry in the `local.oplog.rs` collection. Secondaries continuously fetch and apply new oplog entries.

The `local.oplog.rs` collection is capped: when it reaches its size limit, the oldest entries are deleted. Oplog entries are idempotent — reapplying an operation produces the same result — because they may be applied multiple times on secondaries.

An oplog entry has the following format:

    {
      "ts" : Timestamp(1446011584, 2),
      "h" : NumberLong("1687359108795812092"), 
      "v" : 2, 
      "op" : "i", 
      "ns" : "test.nosql", 
      "o" : { "_id" : ObjectId("563062c0b085733f34ab4129"), "name" : "mongodb", "score" : "100" } 
    }

A secondary performs an initial synchronization (init sync) on first join, copying the full dataset from the primary or a more up-to-date secondary. After that, it uses a tailable cursor to continuously fetch and apply new oplog entries from the `local.oplog.rs` collection of the primary node.

The `init sync` process is as follows:

1. At T1, the secondary copies all databases (except `local`) from the primary using listDatabases, listCollections, and cloneCollection. Assume all operations complete at T2.

2. The secondary applies all oplog entries generated between T1 and T2. Some may overlap with Step 1, but reapplication is safe because oplog entries are idempotent.

3. The secondary creates indexes based on the primary's index settings. The `_id` index for each collection is already created in Step 1.

   Note

   Size the oplog based on your database size and write volume. If the oplog is too large, storage space is wasted. If it is too small, `init sync` may never complete — if the database is large, the oplog might not retain all entries between T1 and T2, causing synchronization to fail.

Modify replica set configuration

To add or remove members, or change properties such as `priority`, `vote`, `hidden`, or `delayed`, use replSetReconfig or rs.reconfig().

For example, to set the second member's priority to 2:

    cfg = rs.conf();
    cfg.members[1].priority = 2;
    rs.reconfig(cfg);

Error Handling (Rollback)

If the primary goes down with unsynced data and writes occur on the new primary before the old one reconnects, the old primary rolls back its unsynced operations to match the new primary's dataset.

Rolled-back data is saved to a rollback directory. Administrators can recover it using mongorestore if needed.

Read and write settings

• Read Preference

  By default, all reads go to the primary. Configure read preference in the driver to route reads to other nodes.

  • primary: The default mode. All reads go to the primary.

  • primaryPreferred: Reads from the primary; falls back to secondaries if the primary is unreachable.

  • secondary: All reads go to secondaries.

  • secondaryPreferred: Reads from secondaries; falls back to the primary if all secondaries are unreachable.

  • nearest: Reads from the nearest reachable node, determined by ping latency.

• Write Concern

  By default, the primary returns a response after completing a write. Configure Write Concern in the driver to define successful write rules.

  The following example requires a write to succeed on a majority of nodes within 5 seconds.

      db.products.insert(
        { item: "envelopes", qty : 100, type: "Clasp" },
        { writeConcern: { w: "majority", wtimeout: 5000 } }
      )

  The preceding method applies to a single request. To set the default write concern for the entire replica set:

      cfg = rs.conf()
      cfg.settings = {}
      cfg.settings.getLastErrorDefaults = { w: "majority", wtimeout: 5000 }
      rs.reconfig(cfg)