Assistant Engineer
Assistant Engineer
  • UID626
  • Fans1
  • Follows1
  • Posts52

[Others]Principle of MongoDB replica set

More Posted time:Nov 29, 2016 9:49 AM
Introduction of replica set
MongoDB replica set is composed of a group of MongoDB instances (processes), including one primary node and multiple secondary nodes. All the data on the MongoDB Driver (client) is written to the primary node, and the secondary node synchronizes the written data from the primary node to ensure that all the members in the replica set store the same data sets to achieve high availability of data.
The figure below (contributed by the official documents of MongoDB) shows a typical MongoDB replica set including a primary node and two secondary nodes.

Primary election
The replica set is initialized through the replSetInitiate command (or rs.initiate() of mongo shell). After the initialization, various members start to send the heartbeat messages and initialize the primary node election. The node that wins the “majority” of votes will become the primary node and other nodes become the secondary nodes.
Initialize the replica set
config = {
    _id : "my_replica_set",
    members : [
         {_id : 0, host : ""},
         {_id : 1, host : ""},
         {_id : 2, host : ""},


Definition of “majority”
Supposing the number of voting members (to be introduced later) in the replica set is N, by “majority”, it means “N/2 + 1”. When the number of members alive in the replica set is less than the “majority”, the replica set cannot elect the primary node, and is not able to provide write services. The replica set is in the read-only status.

We usually recommend you set the number of members in a replica set to an odd number. From the table above, we can see that the replica sets with three nodes and four nodes can both tolerate only one failed node. From the perspective of “service availability”, the effect is the same. (But undoubtedly four nodes can provide more reliable data storage.)
Special secondary node
In normal cases, the secondary node of the replica set will participate in the primary node election (itself may also be elected as the primary node), and synchronize the data last written from the primary node to ensure its data consistency with the primary node.
The secondary node can provide the reading service. Increasing the number of secondary nodes can enhance the reading service capability of the replica set and improve the availability of the replica set. In addition, the MongoDB supports flexible configurations on the secondary nodes in the replica set to adapt to the demands of a variety of scenarios.
The arbiter node only participates in the voting. It cannot be elected as the primary node and does not synchronize data from the primary node.
For example, you deploy a replica set with two nodes, one being the primary node and the other being the secondary node. If either node fails, the replica set will not be able to provide services (cannot elect the primary node). At this time, you can add an arbiter node to the replica set so that even if a node fails, the primary can still be elected.
The arbiter node itself does not store data and is a very lightweight service. When the number of members in the replica set is an even number, you should add an arbiter node to improve the availability of the replica set.
The election priority of a Priority0 node is 0 and the Priority0 node won’t be selected as the primary node.
For example, if you deploy a replica set across server rooms A and B, and want to specify that the primary node must be in server room A, you can set the priority of replica set members in server room B to 0, so that the primary node must be a member in server room A. (Note: If you deploy the replica set like this, you should deploy the “majority” of nodes in server room A. Otherwise the primary node may fail to be elected during network partitioning.)
In MongoDB 3.0, you can set a maximum number of 50 replica set members, and a maximum number of 7 members participating in the primary node election. The vote attributes of other members (Vote0) must be set to 0, that is, they do not participate in the voting.
The hidden node cannot be selected as the primary node (its Priority is 0) and is invisible to the Driver.
Because the hidden node won’t accept requests from the Driver, you can use the hidden node for some data backup, offline computing and other tasks without affecting the service provided by the replica set.
The delayed node must be a hidden node, and its data lags behind that on the primary node for some time (this is configurable, such as one hour).
Because the data on the delayed node lags behind that on the primary node for some time, when incorrect or invalid data is written to the primary node, you can recover the data of a past time point using the data on the delayed node.
Data synchronization
The primary node and the secondary node synchronize data through the oplog. After the write operation is completed on the primary node, the primary node will write an oplog to the special set and the secondary node keeps pulling and applying the oplog from the primary node.
Because the oplog data will keep increasing, the is set to a capped set. When the capacity reaches the upper limit for configuration, it will delete the oldest data. In addition, considering that there may be repeated application of oplog on the secondary node, the oplog must have idempotence so that repeated application of oplog will get the same results.
For example, the oplog format below contains the ts, h, op, ns, and o fields.
  "ts" : Timestamp(1446011584, 2),
  "h" : NumberLong("1687359108795812092"),
  "v" : 2,
  "op" : "i",
  "ns" : "test.nosql",
  "o" : { "_id" : ObjectId("563062c0b085733f34ab4129"), "name" : "mongodb", "score" : "100" }

 ts: Operation time, the current timestamp + counter and the counter is reset every second.
 h: The global unique identifier of the operation.
 v: The oplog version information.
 op: Operation type.
 i: Insert operation.
 u: Update operation.
 d: Delete operation.
 c: Execute commands (such as createDatabase, and dropDatabase)
 n: Null operation. It is for some special purposes.
 ns: The targeted set of the operation.
 o: Operation content. If it is an update operation:
 o2: The operation query condition. Only the update operation contains this field.
When the secondary node initializes the data for the first time, it will first execute init sync to synchronize the full data from the primary node (or other secondary nodes with data updates) and then keep inquiring and applying the latest oplog to itself through tailable cursor from the set of the primary node.
The init sync process contains the following steps:
1. At T1, the secondary node synchronizes the data of all the databases on the primary node (except local) through the sensitive command combination of listDatabases + listCollections + cloneCollection. We suppose all the operations are completed at T2.
2. Apply all the oplogs from the period of [T1-T2] from the primary node. Some operations may have been included in Step 1. But because of the idempotence of the oplog, the oplog can be applied repeatedly.
3. Create indexes for corresponding sets on the secondary node according to the indexing settings of various sets on the primary node. (The _id index of every set has been completed in Step 1.)
The size of the oplog set should be reasonably configured based on the database scale and application writing demands. If the set is too big in size, it will cause a waste of storage space; if the set is too small in size, it may lead to constantly failed init sync of the secondary node. For example, because of the too much data in the database in Step 1 and the too small size of the oplog, the oplog is not enough to store all the oplogs during the period of [T1, T2]. As a result, the secondary node cannot synchronize the data sets from the primary node completely.
Modify replica set configurations
When you need to modify the replica set, such as adding a member, deleting a member or modifying the member configuration (such as priority, vote, hidden, and delayed among other attributes), you can use the replSetReconfig command (rs.reconfig()) to re-configure the replica set.
For example, to set the priority of the second member in the replica set to 2, you can execute the following commands:
cfg = rs.conf();
cfg.members[1].priority = 2;

Details on primary node election
Apart from at the replica set initialization, the primary node election may also occur in the following scenario:
 The replica set is re-configured
 The secondary node will trigger a new round of primary node election when it detects the primary node failure.
 When the primary node performs an active stepDown (actively downgrade to the secondary node), a new round of primary node election will also be triggered.
The primary node election is affected by multiple factors including the inter-node heartbeats, priority and the latest oplog time.
Inter-node heartbeat
Members in a replica set will send a heartbeat message between each other every two seconds by default. If the heartbeat message of a node is not received for 10 seconds, it is deemed that the node has failed; if the failed node is the primary node, the secondary node (the premise is that it can be voted as the primary node) will initiate a new round of primary node election.
Node priority
 Every node is inclined to vote the node with the highest priority as the primary node.
 A node with the priority of 0 won’t take the initiative to trigger the primary node election.
 When the primary node finds there is a secondary node with a higher priority, and the data latency on the secondary node is within 10 seconds, the primary node will perform an active stepDown and make the secondary node with a higher priority eligible for being the primary node.
Only the node with the latest optime (the timestamp of the most recent oplog record) can be elected as the primary node.
Network partition
Only a node that maintains network connection with a majority of nodes is eligible to be elected as the primary node; If the primary node gets disconnected with a majority of nodes, the primary node will take the initiative to downgrade to a secondary node. During network partitioning, multiple primary nodes may appear within a short period of time. So when you write data to the driver, you’d better set the “success for majority” policy so that even if multiple primary nodes appear, only one primary node can successfully write data to the majority of nodes.
Read/write settings of the replica set
Read preference
By default, all the read requests in the replica set are sent to the primary node and the driver can route the read requests to other nodes through setting the Read Preference.
 Primary: The default rule is that all the read requests are sent to the primary node.
 PrimaryPreferred: The primary enjoys priority. If the primary node is unreachable, the requests are sent to the secondary nodes.
 Secondary: All the read requests are sent to the secondary node.
 SecondaryPreferred: The secondary node enjoys priority. When all the secondary nodes are unreachable, the requests are sent to the primary node.
 Nearest: The read requests are sent to the nearest reachable node (detected through the ping).
Write concern
By default, the primary node returns the data as soon as it completes the write operation. The Driver can set the write success rules through setting [Write Concern (].
For example, the write concern rule below sets that the write operation must be successful on a majority of nodes and the timeout value is 5 seconds.
  { item: "envelopes", qty : 100, type: "Clasp" },
  { writeConcern: { w: majority, wtimeout: 5000 } }

The setting above is for a single request. You can also modify the default write concern of the replica set so that you don’t need to set it separately for every single request.
cfg = rs.conf()
cfg.settings = {}
cfg.settings.getLastErrorDefaults = { w: "majority", wtimeout: 5000 }

Exception handling (rollback)
When the primary node is down and the primary node re-joins the set, if some data is not synchronized to the secondary node and there have been some write operations on the new primary node, the old primary node needs to roll back some operations to ensure the consistency of the data set with the new primary node.
The old primary node writes the rollback data to the separate rollback directory and the database administrator can use mongorestore to recover the data as needed.