CPU utilization is a critical monitoring metric for an ApsaraDB for MongoDB instance. Excessively high CPU utilization slows down MongoDB responses and can even make your services unavailable. This topic describes how to view the CPU utilization of an instance, the common causes of high CPU utilization, and the optimization strategies.
How to view CPU utilization
View monitoring charts: On the Monitoring Information page of the ApsaraDB for MongoDB console, you can view the CPU utilization of your ApsaraDB for MongoDB instance. For more information about the collection granularity and the procedure, see basic monitoring.
ApsaraDB for MongoDB provides a variety of node combinations for instance architectures. You can select a node to view its CPU utilization.
Replica set instances A replica set instance consists of a primary node, one or more secondary nodes, a hidden node, and one or more optional read-only nodes.
In the sharded cluster architecture, the CPU usage of each shard is the same as that of a replica set. Configserver only stores configuration metadata and rarely causes a CPU bottleneck, so you can generally ignore it. The CPU utilization of Mongos routing nodes usually depends on aggregation result sets and the number of concurrent requests.
CPU utilization also depends on instance specifications. For example, if an instance is equipped with 8 CPU cores and 16 GB of memory and has a CPU utilization of 100%, 8 CPU cores are exhausted. In this example, the CPU utilization is displayed as 100% instead of 800%.
Common causes
Excessive documents scanned
ApsaraDB for MongoDB supports multi-threading. If a single query needs to scan large numbers of documents, the thread that executes the query occupies CPU resources for a longer period of time. If the pending requests or the queries that need to scan large numbers of documents are in high concurrency, high CPU utilization occurs on the instance on which the queries are executed. The CPU utilization of an instance is positively related to the total number of scanned documents in the instance. Queries that often need to scan large numbers of documents are common in the following scenarios:
Full collection scan
If you find the
COLLSCANkeyword in the slow query log or the system.profile collection, a query performed a full collection scan. You must enable the database profiler to use the system.profile collection. For more information about viewing slow query logs and using the database profiler, see Slow query logs.For more information about how to interpret query execution plans, see Explain Results and Cursor Methods.
Unreasonable index design and usage
Pay close attention to any frequently executed query where the
docsExaminedvalue (number of documents scanned) exceeds 1,000. In addition to a full collection scan, the following situations can lead to a highdocsExaminedvalue:When multiple filter conditions are used, a compound index is not used or the prefix matching principle is not satisfied.
The query is complex or involves large numbers of aggregate operations, which can result in invalid parsing policies or indexes that cannot be optimized.
The data selectivity of a data field and its execution frequency are misjudged, so the best trade-off is not reached.
Excessive concurrency
If large numbers of service requests are sent and the concurrency is excessively high, the CPU utilization is high. To solve this high CPU utilization issue, you can add CPU cores if you confirm that no query issues occur.
Although an instance supports a high maximum total number of connections, its capacity to process active connections is limited by its CPU cores. When the number of active connections exceeds the number of CPU cores, CPU resource contention can cause thread queuing and task backlogs. This can result in slow responses even when monitoring reports a low overall CPU utilization.
Other causes
Large numbers of short-lived connections are established. In versions later than MongoDB 3.X, the default identity authentication mechanism is SCRAM-SHA1 that requires CPU-intensive operations such as hash calculation. If short-lived connections are in high concurrency, hash calculation consumes multifold CPU resources and even exhausts CPU resources. In this case, operational logs contain large numbers of saslStart error messages. To optimize PHP short-lived connections in high concurrency scenarios, ApsaraDB for MongoDB optimizes the method to rewrite built-in random functions at the kernel layer. This helps reduce the CPU utilization on an instance.
Time-to-live (TTL) indexes cause the higher CPU utilization of a secondary node than the primary node. In this case, you can ignore the high CPU utilization of the node.
MongoDB 3.2 and later support multi-threaded replication. The replay concurrency of oplogs is determined by the replWriterThreadCount parameter. The default value of this parameter is 16. Secondary nodes do not handle any service writes. However, in some scenarios the CPU utilization of a secondary node may exceed that of the primary node. For example, if you configure TTL-based automatic deletion for a collection on the primary node, the system deletes data in batches based on the index on the time column, which is efficient. The system converts that operation into many single delete operations and sends them to the secondary node. On the secondary node, replaying oplogs is less efficient, so multi-threaded replay can easily raise the CPU utilization of that node.
TTL deletions can remove thousands of expired documents per second. The resulting rise in CPU utilization and disk I/O is expected load. Scale up the instance when the workload requires more capacity, or wait until fragmented-data cleanup is complete and then enable the balancer to rebalance data.
For a sharded cluster, investigate further if CPU utilization does not decrease and insert performance does not improve after orphaned documents are cleaned up. In CloudDBA or monitoring metrics, compare the dirty cache and TTL deletion rate of each shard. A high dirty cache can cause application threads to participate in flushing dirty pages, and an unusually high TTL deletion rate on one shard identifies an imbalanced delete workload.
Troubleshooting methods
View and terminate active sessions
If the sessions of a normally running instance suddenly surge to 100%, the cause is usually a change on the application side. Common causes include scanning too many documents, data sorting and aggregation, and a spike in service traffic. You can troubleshoot the issue as follows:
In the ApsaraDB for MongoDB console, go to the page to view active sessions. Analyze any queries that are taking longer than expected and terminate the active session or take other corrective actions.
To view and analyze the details of active sessions, run the db.currentOp() command provided by MongoDB. If necessary, run the db.killOp() command to actively terminate slow queries that are not completed within the expected execution period. For more information, see db.currentOp() and db.killOp().
Record and view logs
When CPU utilization increases abnormally, you can analyze the slow query log or audit log to identify problematic requests. Check for keywords such as COLLSCAN and docsExamined to confirm whether an excessive number of documents are being scanned.
Audit logs
In the ApsaraDB for MongoDB console, go to the page to enable and view the audit log. For more information about how to enable and use this feature, see Enable the audit log feature.
Slow query logs
ImportantYou can view only the slow query logs of the last seven days.
For instances purchased after June 6, 2021, you must first enable the audit log feature and set the audited operation types to include admin and slow. You can then view the slow query log. Only slow queries that occur after the audit log is enabled are recorded.
In the ApsaraDB for MongoDB console, go to the page. Configure the operationProfiling.mode (the slow query mode) and operationProfiling.slowOpThresholdMs (the slow query threshold) parameters based on your business needs.
MongoDB provides three profiling modes:
Profiling is disabled and no data is collected.
Profiling is enabled for all requests. The execution data of all requests is recorded in the system.profile collection.
Profiling is enabled for slow queries. Queries that take longer than the specified threshold are recorded in the system.profile collection.
For more information about the profiling parameters, see Database Profiler.
On the page, view the slow query logs.
Optimization strategies
Optimize indexes
Index optimization is the best solution to reduce the number of documents that a single query needs to scan. In the underlying architecture, ApsaraDB for MongoDB uses an index design similar to MySQL and provides richer categories and features than MySQL. Therefore, most index optimization policies that apply to MySQL also apply to ApsaraDB for MongoDB.
For more information about how to create and use indexes, see Best practices for creating indexes in ApsaraDB for MongoDB or the following official documents:
For more information about compound indexes, see Compound Indexes.
For more information about how to use indexes to sort query results, see Use Indexes to Sort Query Results.
For more information about hints, see Cursor Methods and cursor.hint().
For more information about how to balance the data selectivity of a data field with the selection frequency, see Create Queries that Ensure Selectivity.
Add CPU cores
If you confirm that no query issues occur, the high CPU utilization is caused by the high number of service requests and the high concurrency. You can add CPU cores to solve the high CPU utilization issue. In most case, you can use one of the following methods to add CPU cores:
Scale up a single instance for more read and write workloads.
Configure read/write splitting at the replica set level, or add read-only instances to the replica set.
Upgrade the problematic instance to a sharded cluster instance for linear scale-out.
If the CPU of Mongos routing nodes is exhausted, add Mongos nodes and configure load balancing for them. For more information about load balancing of Mongos nodes, see Load balancing.
For more information about how to change the configurations of an ApsaraDB for MongoDB instance, see Change the configuration of a standalone instance, Change replica set instance configurations, and Change the configurations of a sharded cluster instance.
Control the number of collections and the execution frequency
We recommend that you add indexes first to optimize full collection scans. If this method no longer helps, control the data volume of collections and the execution frequency on the application side.
Avoid frequent short-lived connections
We recommend that you use persistent connections whenever possible.
FAQ
Why does CPU utilization increase after a MongoDB version upgrade?
A version upgrade can increase CPU utilization when default parameters change or a query plan changes. Use the following steps to identify the affected operations and optimize them.
Run
db.currentOp()inmongoshto identify long-running or abnormal requests. If an abnormalopidis returned, rundb.killOp(opid)to terminate that operation.In the ApsaraDB for MongoDB console, go to the CloudDBA > Instance Sessions page to analyze active sessions and find requests with an unexpected execution time.
Go to the Log Management > Slow log page to identify collection scans or queries that do not use indexes. Optimize indexes based on the slow query log results.