Overview of new features in MongoDB 5.0 - ApsaraDB for MongoDB

Background information

MongoDB 5.0 marks the beginning of a new release cycle to deliver new features to users faster. For example, features such as the Versioned API and live resharding simplify future database upgrades and accommodate evolving business needs. The native time series data platform enables MongoDB to support a wider range of workloads and use cases, and the new MongoDB Shell enhances the user experience.

Native time series platform

MongoDB 5.0 extends its general-purpose application data platform to simplify handling time series data, expanding its use cases in areas like IoT, financial analytics, and logistics.

MongoDB time series collections automatically store time series data in a highly optimized and compressed format, reducing storage size and I/O for better performance and scale. This shortens the development cycle and lets you quickly build models optimized for time series applications.

Example of a command to create a time series collection:

db.createCollection("collection_name",{ timeseries: { timeField: "timestamp" } } )

MongoDB can seamlessly adjust ingestion frequency and automatically handle out-of-order measurements based on dynamically generated time partitions. The latest MongoDB Connector for Apache Kafka provides native support for time series. You can automatically create a time series collection directly from Kafka topic messages, which lets you process and aggregate data as you ingest it before writing the data to a MongoDB time series collection.

A time series collection automatically creates a time-ordered clustered index, which reduces query latency. The MongoDB query API is also extended with window functions, allowing you to run analytical queries such as moving averages and cumulative sums. In a relational database system, these are often called SQL analytical functions and support windows defined by rows (for example, a three-row moving average). MongoDB goes a step further by adding powerful time series functions like exponential moving average (EMA), derivatives, and integrals, which support windows defined by time (for example, a 15-minute moving average). You can use window functions to query both time series and regular MongoDB collections, providing new analytical capabilities for various application types. Additionally, MongoDB 5.0 provides new date operators, including $dateAdd, $dateSubtract, $dateDiff, and $dateTrunc, for aggregating and querying data over custom time windows.

You can locate a time series collection in the same database as a regular MongoDB collection. You do not need to choose a specialized time series database, which cannot serve any other type of application, or implement complex integrations to mix time series and other data. By providing a unified platform, MongoDB lets you build high-performance, efficient time series applications while also supporting other use cases or workloads. This eliminates the cost and complexity of integrating and running multiple, disparate databases.

Live resharding

Database version	Features	Implementation method
Before MongoDB 5.0	Before MongoDB 5.0, resharding was a complex, manual process.	Method 1: Dump the entire collection and then reload the data into a new collection with the new shard key. Because this is an offline process, your application experiences significant downtime until the reload is complete. For example, dumping and reloading a collection of over 10 TB on a three-shard cluster could take several days. Method 2: Create a new sharded cluster, redefine the collection's shard key, and then use a custom migration process to write the data from the old sharded cluster into the new one. This process requires you to handle query routing and migration logic yourself, and continuously check the migration progress to ensure all data migrates successfully. Custom migration is a highly complex, labor-intensive, risky, and time-consuming task. For example, one MongoDB user spent three months migrating 10 billion documents.
MongoDB 5.0 and later	reshardCollectionRun the command to start resharding. The resharding process is highly efficient. It does not simply rebalance data. Instead, it copies and rewrites all data from the current collection to a new collection in the background, while staying in sync with new application writes. Resharding is fully automated. This reduces resharding time from weeks or months to minutes or hours, avoiding tedious and complex manual data migrations. With live resharding, you can easily evaluate the effects of different shard keys in a development or test environment and change your shard key whenever you need to.	reshardCollectionYou can change a collection's shard key on demand as your business grows, without database downtime or complex data migrations. You only need to run the command in the MongoDB Shell, select the database and collection to reshard, and then specify the new shard key. `reshardCollection: "<database>.<collection>", key: <shardkey>` Note <database>: The name of the database to reshard. <collection>: The name of the collection to reshard. <shardkey>: The name of the shard key. reshardCollectionAfter applying all oplog entries, MongoDB automatically switches to the new collection and deletes the old one in the background.

Versioned API

Application compatibility
Starting with MongoDB 5.0, the Versioned API defines a stable set of the most common commands and parameters, which remain unchanged across both annual Major Releases and quarterly Rapid Releases. By decoupling the application and database lifecycles, you can pin your driver to a specific version of the MongoDB API. This ensures your application can run for years without code changes, even through database upgrades and improvements.
Flexibility to add new features and improvements
The Versioned API allows MongoDB to flexibly add new features and improvements to the database in each release while maintaining backward compatibility. When you need to change the API, you can introduce a new API version to run on the same server alongside existing versioned APIs. As MongoDB accelerates its release schedule, the Versioned API enables you to adopt the latest MongoDB features faster and more easily.

Write Concern defaults to the Majority level

Starting with MongoDB 5.0, the default write concern level is majority. A write operation is acknowledged as successful only after it has been applied to the primary node and persisted to the journal of a majority of secondary nodes, providing stronger data durability guarantees out of the box.

Note The write concern is fully tunable. You can customize the write concern to balance application performance and data durability.

Connection management optimizations

By default, each client connection corresponds to one thread on the backend MongoDB server when you configure net.serviceExecutor as synchronous. Creating, switching, and destroying threads are resource-intensive operations. When the number of connections is too high, threads consume significant server resources.

A connection storm is a high number of connections or an uncontrolled rate of connection creation. This issue can have various causes and often occurs when service performance is already degraded.

To address these situations, MongoDB 5.0 introduces the following measures:

It limits the number of connections that drivers can create at one time, providing a simple and effective way to prevent database server overload.
It reduces how often drivers check the connection pool, giving unresponsive or overloaded server nodes time to recover.
Drivers direct workloads to faster servers with healthier connection pools instead of randomly selecting from available servers.

These measures, combined with improvements to the mongos query routing layer from previous versions, further enhance MongoDB's ability to handle high-concurrency loads.

Long-running snapshot queries

Long-running snapshot queries increase application versatility and resilience. You can run queries that default to a 5-minute duration (or adjust it to a custom length) while maintaining snapshot isolation consistent with a live transactional database. You can also perform snapshot queries on a secondary node, which allows you to run different workloads on a single cluster and scale them out across different shards.

MongoDB implements long-running snapshot queries through a feature in the underlying storage engine called Durable History, which was introduced in MongoDB 4.4. Durable History stores a snapshot of all field values that have changed since a query began. By using Durable History, queries can maintain snapshot isolation even as data changes. Durable History also helps reduce cache pressure on the storage engine, enabling higher query throughput in scenarios with high write loads.

New MongoDB Shell

To provide a better user experience, MongoDB 5.0 introduces the completely redesigned MongoDB Shell (mongosh). It offers a modern command-line experience, enhanced usability features, and a powerful scripting environment. The new MongoDB Shell is now the default shell for the MongoDB platform. It includes features like syntax highlighting, intelligent autocompletion, contextual help, and helpful error messages for an intuitive and interactive experience.

Enhanced user experience
- Write queries and aggregations more easily, and get more readable results.
  The new MongoDB Shell supports syntax highlighting, making it easier to distinguish between fields, values, and data types to avoid syntax errors. If an error occurs, the new MongoDB Shell pinpoints the problem and suggests a solution.
- Enter queries and commands faster.
  The new MongoDB Shell supports intelligent autocompletion. It provides suggestions for methods, commands, and MQL expressions based on the MongoDB version you are using.
  Example: If you cannot remember the syntax for a command, you can quickly look it up directly from the MongoDB Shell.
```
secondary [direct: secondary] sample_mflix → db.movies.createIndex
db.movies.createIndexes  db.movies.createIndex
secondary [direct: secondary] sample_mflix → db.movies.createIndex.help()
  db.coll.createIndex({ category: 1 }, { name: 'index-1' }):
  Creates one index on a collection
  For more information on usage: https://docs.mongodb.com/manual/reference/method/db.collection.createIndex
secondary [direct: secondary] sample_mflix →
```
Advanced scripting environment
The new MongoDB Shell's scripting environment is built on the Node.js REPL (Read-Eval-Print Loop). You can use all Node.js APIs and any modules from npm in your scripts. You can also load and run scripts from the file system, and you can continue to use load() and eval() to execute scripts just as in the legacy shell.
Extensibility and plugins
The new MongoDB Shell is highly extensible, allowing you to use all of MongoDB's features to improve your productivity.

In the new MongoDB Shell, you can install Snippets plugins. The shell can automatically load Snippets and can use all Node.js APIs and npm packages. MongoDB also maintains a Snippets repository with useful features, such as a plugin for analyzing the schema of a specified collection. You can also configure the MongoDB Shell to use plugins of your choice.
Note Plugins are currently an experimental feature in the MongoDB Shell.

PyMongoArrow and data science

With the new PyMongoArrow API, you can use Python to run complex analytics and machine learning on MongoDB. PyMongoArrow can quickly convert simple MongoDB query results into popular data formats, such as a Pandas DataFrame or NumPy array, to streamline your data science workflows.

Schema validation improvements

Schema validation is a way to enforce data governance controls in MongoDB. In MongoDB 5.0, schema validation is simpler and more user-friendly. When an operation fails validation, MongoDB generates a descriptive error message. This message helps you understand which document did not conform to the collection's validator rules and why, allowing you to quickly identify and correct the code that violates the rules.

Resumable index builds

In MongoDB 5.0, an in-progress index build automatically resumes from where it left off after a node restart. This reduces the business impact of planned maintenance actions. For example, when you restart or upgrade a database node, you no longer need to worry about an ongoing index build on a large collection failing.

Release model changes

Because MongoDB supports many versions and platforms, each release must be validated on more than 20 supported platforms. This extensive validation work slowed down the delivery of new features. Therefore, starting with MongoDB 5.0, releases are divided into Major Releases and Rapid Releases. Rapid Releases are available for download for development and testing purposes, but they are not recommended for production environments.

More features

For more information about the features of MongoDB 5.0, see Release notes for MongoDB 5.0.