ApsaraDB for MongoDB:MongoDB 4.4 features

Hidden indexes

Maintaining too many indexes degrades write performance. However, operational complexity often makes administrators reluctant to remove a potentially inefficient index, fearing that an incorrect deletion could cause performance jitter. Rebuilding an index is also a costly operation.

To address this challenge, ApsaraDB for MongoDB and MongoDB jointly developed the hidden index feature as part of their strategic partnership. This feature lets you hide an index using the collMod command, which prevents the query planner from using it. After an observation period confirms that there are no negative impacts on your application, you can safely delete the index.

db.runCommand( {
   collMod: 'testcoll',
   index: {
      keyPattern: 'key_1',
      hidden: false
   }
} )

Note that hiding an index only makes it invisible to the MongoDB query planner. It does not change the index's special behaviors, such as unique index constraints or TTL expiration.

Refinable shard keys

In a MongoDB sharded cluster, a well-designed shard key is crucial for achieving good scalability under a specific workload. However, in practice, even a carefully chosen shard key can become problematic as the workload changes, leading to jumbo chunks (chunks that exceed a preset size) or causing most of the traffic to hit a single shard.

In MongoDB 4.0 and earlier, a collection's shard key and its corresponding values were immutable. Version 4.2 let you modify a shard key's value, but the process involved cross-shard data migration based on distributed transactions, incurring high performance overhead without fully solving the problem of jumbo chunks or query hotspots. For example, consider an orders table with the shard key {customer_id:1}. This key may be sufficient when the business is new and each customer has few orders. As the business grows, a major customer might accumulate a large number of orders, turning that customer's data into a query hotspot on a single shard. Because orders are inherently tied to a customer_id, simply modifying the customer_id value does not resolve the issue of uneven access.

In such scenarios, MongoDB 4.4 lets you use the refineCollectionShardKey command to add one or more suffix fields to an existing shard key. This helps improve the distribution of documents across chunks. In the order processing scenario described above, you can use the refineCollectionShardKey command to change the shard key to {customer_id:1, order_id:1}, which prevents query hotspots on a single shard.

The refineCollectionShardKey command has very low performance overhead because it only modifies metadata on the config server nodes and does not require any data migration. Data redistribution occurs gradually through normal automatic chunk splitting and migration. A shard key must be supported by a corresponding index, so the refineCollectionShardKey command requires you to create an index for the new shard key beforehand.

Because not all documents may contain the new suffix fields, MongoDB 4.4 implicitly supports missing shard key fields. This means newly inserted documents do not need to include all fields of the shard key. However, this practice is not recommended, as it can easily lead to jumbo chunks.

Compound hashed shard keys

Before version 4.4, you could only specify a single-field hashed shard key because MongoDB did not support compound hashed indexes. This often led to an uneven distribution of collection data across shards.

MongoDB 4.4 introduces support for compound hashed indexes. You can now specify a single hashed field within a compound index. This field can appear in any position, as a prefix or a suffix, which in turn enables support for compound hashed shard keys.

Example:

sh.shardCollection(
  "examples.compoundHashedCollection",
  { "region_id" : 1, "city_id": 1, field1" : "hashed" }
)
sh.shardCollection(
  "examples.compoundHashedCollection",
  { "_id" : "hashed", "fieldA" : 1}
)

Without support for compound hashed shard keys, the only solution was to pre-calculate a hash of the required field, store the result in a separate field within the document, and then use ranged sharding on that field.

In version 4.4, you can solve this problem by simply specifying the target field as hashed. For the second scenario, setting the shard key to {customer_id:'hashed', order_id:1} greatly simplifies your application logic.

Hedged reads

Slow page response times can cause financial loss. A research report from Google shows that if a page takes more than three seconds to load, the bounce rate increases by 50%. To address this, MongoDB 4.4 introduces hedged reads. In a sharded cluster, a mongos node can send a read request to two replica set members of a shard and return the first response. This helps reduce P95 (95th percentile) and P99 (99th percentile) latencies for your application.

db.collection.find({ }).readPref(
   "secondary",                      // mode
   [ { "datacenter": "B" },  { } ],  // tag set
   { enabled: true }                 // hedge options
)
)

To use hedged reads, you must also enable support on the mongos node by setting the readHedgingMode parameter to on.

db.adminCommand( { setParameter: 1, readHedgingMode: "on" } )

Reduced replication latency

MongoDB 4.4 reduces primary/secondary replication latency. In MongoDB, this latency can significantly affect read and write operations. In certain scenarios, a secondary node must replicate and apply incremental updates from the primary node promptly to continue processing reads and writes. Therefore, lower replication latency provides better consistency.

Streaming replication

In versions before 4.4, a secondary node had to continuously poll an upstream source to fetch incremental updates. In each polling cycle, the secondary sent a getMore command to the primary to read the Oplog. If data was available, the primary returned a batch of up to 16 MB. If not, the secondary used the awaitData option to reduce unnecessary getMore overhead while still being able to fetch new Oplog entries as soon as they appeared. A single OplogFetcher thread handled this pull operation, and each batch fetch required a full round-trip time (RTT). In replica sets with poor network conditions, network latency severely limited replication performance.

In version 4.4, incremental Oplog entries are continuously streamed to secondary nodes instead of being pulled. Compared to the polling method, this saves at least half of the RTT for Oplog fetching. Streaming replication significantly improves performance in the following two scenarios:

• When a user's write operation specifies a writeConcern of "majority", the operation must wait for acknowledgments from a majority of replica set members. With the new replication mechanism, the performance of majority writes can improve by an average of 50%, even in high-latency network environments.

• When a user employs causal consistency to ensure "read your writes" guarantees, the application relies on the timely replication of Oplog entries from the primary to the secondary nodes.

Simultaneous indexing

Before version 4.4, the primary node had to fully build an index before secondary nodes could begin the process. The method of index creation on secondaries varied across versions, with different impacts on the Oplog.

Even in version 4.2, which unified foreground and background index builds and used fine-grained locking (exclusive locks were held only at the beginning and end of the build), the CPU and I/O overhead of index creation could still cause replication latency. Certain operations, such as modifying collection metadata with the collMod command, could block Oplog application. In worst-case scenarios, a secondary could fall so far behind that Oplog entries on the primary were overwritten, forcing the secondary into a Recovering state.

In version 4.4, index builds occur simultaneously on the primary and secondary nodes. This dramatically reduces the risk of replication latency from this cause and ensures that secondary nodes can access the latest data even during an index build.

Furthermore, the new index build mechanism requires that a majority of voting nodes successfully complete the build before the index becomes usable. This also helps reduce performance discrepancies in read/write splitting scenarios that can be caused by nodes having different indexes available.

Mirrored reads

A common pattern observed in ApsaraDB for MongoDB is that many users with three-node replica set instances direct all read and write operations to the primary node. One of the secondary nodes remains idle, carrying no read traffic. During an occasional failover, users experience a noticeable increase in application latency, which only returns to normal after some time. This happens because the newly elected primary node has not served reads before, so its cache is cold. It does not know the application's access patterns and has not cached the relevant data. As a result, read operations trigger a large number of cache misses, requiring disk reads and increasing access latency. This problem is especially pronounced on instances with large amounts of memory.

To solve this problem, MongoDB 4.4 introduces the mirrored read feature. The primary node can mirror a configurable portion of its read traffic to a secondary node to help pre-warm the secondary's cache. This is a non-blocking, "fire-and-forget" action that has no tangible performance impact on the primary node, although it does slightly increase the load on the secondary node.

The percentage of traffic to mirror is dynamically configurable through the mirrorReads parameter. By default, 1% of the traffic is mirrored.

Example:

db.adminCommand( { setParameter: 1, mirrorReads: { samplingRate: 0.10 } } )

You can also view statistics related to mirrored reads by using the db.serverStatus( { mirroredReads: 1 } ) command, as shown below:

SECONDARY> db.serverStatus( { mirroredReads: 1 } ).mirroredReads
{ "seen" : NumberLong(2), "sent" : NumberLong(0) }

Resumable initial sync

In versions before 4.4, if a network fluctuation caused a connection to drop while a secondary node was performing an initial sync, the secondary had to restart the entire process from the beginning. For large datasets, this could waste hours and significantly impact operations.

In version 4.4, MongoDB allows a secondary node to resume an initial sync process from where it was interrupted. If the connection cannot be re-established after a transient error, the system selects a new sync source and starts a new initial sync. The default timeout for retry attempts is 24 hours, which you can change at startup by using the replication.initialSyncTransientErrorRetryPeriodSeconds parameter.

Note that for interruptions caused by non-transient errors during the initial sync, the full sync process must still be restarted from the beginning.

Time-based oplog retention

The Oplog in MongoDB records all data modification operations. It is used not only for replication but also for scenarios like incremental backups, data migration, and data subscriptions, making it a critical part of the MongoDB data ecosystem.

Most application scenarios require retaining the Oplog for a specific period. However, determining how much Oplog data will be generated during that period is difficult.

In version 4.4, MongoDB allows you to define a minimum retention period for Oplog entries by using the storage.oplogMinRetentionHours parameter. You can also change this value online by using the replSetResizeOplog command. Example:

// First, show current configured value
db.getSiblingDB("admin").serverStatus().oplogTruncation.oplogMinRetentionHours
// Modify
db.adminCommand({
  "replSetResizeOplog" : 1,
  "minRetentionHours" : 2
})

Union

For multi-collection queries, versions before 4.4 only offered the $lookup stage, which is similar to a left outer join in SQL. Version 4.4 introduces the $unionWith stage, which functions like SQL's union all. It combines data from two or more collections into a single result set that you can then query and filter further. Unlike the $lookup stage, the $unionWith stage supports sharded collections. You can use multiple $unionWith stages in an aggregation pipeline to aggregate data from multiple collections. The syntax is as follows:

{ $unionWith: { coll: "<collection>", pipeline: [ <stage1>, ... ] } }

You can also specify a pipeline within the $unionWith stage to filter or transform the data from the other collection before the union. This provides great flexibility. For example, imagine a business stores its order data in separate collections for each month. The data for the second quarter looks like this:

db.orders_april.insertMany([
  { _id:1, item: "A", quantity: 100 },
  { _id:2, item: "B", quantity: 30 },
]);
db.orders_may.insertMany([
  { _id:1, item: "C", quantity: 20 },
  { _id:2, item: "A", quantity: 50 },
]);
db.orders_june.insertMany([
  { _id:1, item: "C", quantity: 100 },
  { _id:2, item: "D", quantity: 10 },
]);

Suppose you need to list the total sales for each product in the second quarter. Before version 4.4, you would have had to read all the data into your application and perform the aggregation there, or rely on a data warehouse. In version 4.4, you can solve this with a single aggregation query:

db.orders_april.aggregate( [
   { $unionWith: "orders_may" },
   { $unionWith: "orders_june" },
   { $group: { _id: "$item", total: { $sum: "$quantity" } } },
   { $sort: { total: -1 }}
] )

Custom aggregation expressions

Before version 4.4, you could execute custom JavaScript on the server to perform complex queries by using the $where operator in the find command or by using the MapReduce feature. However, these features were not integrated with the aggregation pipeline.

In version 4.4, MongoDB introduces the $accumulator and $function operators for the aggregation pipeline. These operators replace the $where operator and MapReduce. They allow you to define a custom aggregation expression using server-side JavaScript. This consolidates complex query functionality into the aggregation pipeline, which improves API consistency and user experience while leveraging the pipeline's execution model.

The $accumulator operator is similar to MapReduce. It first defines an initial state using an init function, then updates the state for each input document using a specified accumulate function. If necessary, it also executes a merge function.

For example, if you use the $accumulator operator on a sharded collection, the results from different shards must be merged. If a finalize function is specified, it will be applied after all input documents are processed to transform the state into the final output.

The $function and $where operator have almost the same functionality. However, the $function operator is more powerful because it can be used with other aggregation pipeline operators. In addition, you can use the $function operator in the find command with the $expr operator. This is equivalent to the $where operator. In its official documentation, MongoDB also recommends that you prioritize using the $function operator.

Other usability enhancements

Connection monitoring and pooling

The drivers for MongoDB 4.4 add capabilities for monitoring and configuring client-side connection pool behavior. You can use standard APIs to subscribe to connection pool-related events, such as connections being opened or closed, and the pool being cleared. You can also use APIs to configure connection pool behavior, such as the maximum and minimum number of connections, the maximum idle time for each connection, and the timeout for a thread waiting for an available connection. For more details, see the MongoDB official documentation.

Global read and write concerns

In versions before 4.4, if an operation did not explicitly specify a readConcern or writeConcern, MongoDB applied a default behavior. For example, readConcern defaulted to local, and writeConcern defaulted to {w: 1}. You could not change this default behavior. If you wanted all insert operations to default to a writeConcern of {w: "majority"}, you had to explicitly specify it in your application code for every operation.

In version 4.4, you can use the setDefaultRWConcern command to configure the global default readConcern and writeConcern. Example:

db.adminCommand({
  "setDefaultRWConcern" : 1,
  "defaultWriteConcern" : {
    "w" : "majority"
  },
  "defaultReadConcern" : { "level" : "majority" }
})

You can also use the getDefaultRWConcern command to retrieve the current default readConcern and writeConcern.

New MongoDB Shell (beta)

function topActors(howMany = 5) {
  return db.movies.aggregate([
    { $unwind: '$cast'},
    { $group: {_id: '$cast', movieCount: { $sum: 1 } } },
    { $sort: {movieCount: -1} },
    { $limit: howMany }
  ])
}
> topActors(3)
[
  { _id: 'Gérard Depardieu', movieCount: 68 },
  { _id: 'Robert De Niro', movieCount: 60 },
  { _id: 'Michael Caine', movieCount: 53 }
]
> db.movies.fnd()
TypeError: db.movies.fnd is not a function
> db.movies.find.help()
  db.collection.find(query, projection):
  Selects documents in a collection or view.
  For more information on usage: https://docs.mongodb.com/manual/reference/method/db.collection.find
> db.movies.find({year: {$gt: 2016}})

Conclusion

The 4.4 release is primarily a maintenance version that brings many enhancements. In addition to the features discussed, there are many smaller optimizations, such as improvements to $indexStats, support for TCP Fast Open to speed up connection establishment, and optimized index deletion. There are also larger enhancements, like the new structured logging format LogV2 and new security mechanisms. For more details, see the official Release Notes.