Problem description
The CPU utilization of your ApsaraDB RDS for MySQL or MariaDB instance is high, sometimes reaching 100%.
Causes
When an application submits a query or data modification, the system performs many logical read operations. These operations involve accessing many data rows in a table. The system then uses significant CPU resources to maintain data consistency between the storage and memory. This topic explains two common causes for 100% CPU utilization and their solutions: high application load, measured in queries per second (QPS), and high query costs from slow SQL statements. High CPU utilization in MySQL is often caused by slow SQL statements that access many data rows, resulting in high query costs.
This topic does not discuss high CPU utilization caused by row lock conflicts, lock waits, or background tasks. These issues occur infrequently.
High application load (high QPS):
Characteristics: The instance has a high QPS. Queries are simple and efficient. There is little room for optimization.
Symptoms: No slow queries occur, or slow queries are not the main cause. The QPS and CPU utilization curves match.
Common scenarios: This issue is common in optimized online transaction processing (OLTP) systems such as order systems, popular web applications with high read rates, and third-party stress testing tools such as Sysbench.
High query costs from slow SQL statements (many data rows accessed):
Characteristics: The instance has a low QPS. Query execution is inefficient and requires scanning large amounts of table data. There is significant room for optimization.
Symptoms: Slow queries exist. The QPS and CPU utilization curves do not match.
Cause analysis: Inefficient queries need to access large amounts of data to retrieve the expected results. This leads to high average logical I/O. As a result, CPU utilization can be high even when the QPS is low, such as on a website with low traffic.
Solutions
Choose a solution based on your specific situation.
High application load (high QPS)
If high CPU utilization is caused by a high application load, there is little room for SQL optimization. Address the issue by adjusting the application architecture or instance type. Consider the following methods:
Upgrade the instance type to add more CPU resources. For more information, see Change instance configurations.
Add read-only instances. Transfer queries that are not sensitive to data consistency, such as product category or train schedule queries, to the read-only instances. This reduces the load on the primary instance. For more information, see Create an ApsaraDB RDS for MySQL read-only instance.
Use PolarDB-X, a cloud-native distributed database from Alibaba Cloud. It automatically performs sharding to distribute the query load across multiple RDS instances.
Use Alibaba Cloud Memcache or Tair (Redis OSS-compatible). Retrieve frequently used query results from the cache to reduce the load on the RDS instance.
For applications that query relatively static data, have high query repetition, and have result sets smaller than 1 MB, consider enabling the query cache.
NoteTest whether enabling the query cache is beneficial for your application. For more information about the settings, see Set and use the query cache for ApsaraDB RDS for MySQL.
Periodically archive historical data. Use sharding or partitioning to reduce the amount of data accessed by queries. Optimize queries to reduce execution costs and improve application scalability.
High query costs from slow SQL statements
The principle for solving this issue is to locate inefficient queries, optimize their execution efficiency, and reduce their execution costs.
Locate inefficient queries in the following ways:
Execute the following SQL statement to view currently running queries.
show processlist; show full processlist;The system displays output similar to the following:
mysql> show processlist; +----------+-------+-------------------+---------+---------+-------+--------------+----------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +----------+-------+-------------------+---------+---------+-------+--------------+----------------------------------------------+ |101031643 | jacky | xxx.xxx.xxx.xx:xx | my_test | Query | 10760 | Sending data | select b.* from perf_test_no_idx_01 a, ... | |117731567 | jacky | xxx.xxx.xxx.xx:xx | my_test | Query | 10760 | Sending data | select b.* from perf_test_no_idx_01 a, ... | |134298793 | jacky | xxx.xxx.xxx.xx:xx | my_test | Query | 10760 | Sending data | select b.* from perf_test_no_idx_01 a, ... | |134384670 | jacky | xxx.xxx.xxx.xx:xx | my_test | Query | 0 | Init | show processlist | |234891284 | jacky | xxx.xxx.xxx.xx:xx | my_test | Query | 10760 | Sending data | select b.* from perf_test_no_idx_01 a, ... | |235125098 | jacky | xxx.xxx.xxx.xx:xx | my_test | Query | 10760 | Sending data | select b.* from perf_test_no_idx_01 a, ... | |235200576 | jacky | xxx.xxx.xxx.xx:xx | my_test | Query | 10760 | Sending data | select b.* from perf_test_no_idx_01 a, ... | |235633985 | jacky | xxx.xxx.xxx.xx:xx | my_test | Query | 10760 | Sending data | select b.* from perf_test_no_idx_01 a, ... | |235887773 | jacky | xxx.xxx.xxx.xx:xx | my_test | Query | 10760 | Sending data | select b.* from perf_test_no_idx_01 a, ... | |251990394 | jacky | xxx.xxx.xxx.xx:xx | my_test | Query | 10760 | Sending data | select b.* from perf_test_no_idx_01 a, ... | |252662718 | jacky | xxx.xxx.xxx.xx:xx | my_test | Query | 10760 | Sending data | select b.* from perf_test_no_idx_01 a, ... | +----------+-------+-------------------+---------+---------+-------+--------------+----------------------------------------------+ 11 rows in set (0.00 sec)Query sessions that run for a long time and have a state of Sending data, Copying to tmp table, Copying to tmp table on disk, Sorting result, or Using filesort may have performance issues.
In scenarios where high QPS causes high CPU utilization, queries often execute quickly. This can make it difficult to capture them with the
show processlist;command or by viewing instance sessions. In this case, run the following SQL statement:explain [$SQL]Note[$SQL] is the SQL query that has performance issues.
You can run a command such as
kill [$ID];to stop a long-running session. For more information about stopping sessions, see How to stop sessions on an ApsaraDB RDS for MySQL instance.Note[$ID] is the session ID that corresponds to the query.
View currently running queries using Database Autonomy Service (DAS):
Log on to the DAS console.
In the navigation pane on the left, click .
Find the target instance and click the instance ID to open the instance details page.
In the left-side navigation pane, click Instance Sessions.
Click the query text in the SQL column to display the full query and its execution plan.
After you identify the queries that need optimization, use SQL Diagnostics in the DMS console to obtain optimization suggestions. The diagnostic report is also useful for troubleshooting historical issues of high CPU utilization:
Log on to the instance in the DMS console. For more information, see Log on to a database instance.
At the top of the page, click SQL Window and select the target database.
Paste the query into the SQL window and click SQL Diagnostics to obtain optimization suggestions.
Follow the optimization suggestions as needed. For example, adding an index can significantly reduce query execution costs.
More information
Features for troubleshooting performance issues
Data Management (DMS) provides several features to help you troubleshoot and resolve instance performance issues. The diagnostic report is the best tool for troubleshooting performance issues on MySQL and MariaDB instances. Regardless of the cause, check the diagnostic report first. Pay special attention to the SQL optimization, session list, and slow SQL summary sections.
Principles for avoiding 100% CPU utilization
Follow these principles to avoid 100% CPU utilization:
Set CPU utilization alerts to ensure your instance has a sufficient CPU buffer.
During application design and development, consider query optimization. Follow the general optimization principles for MySQL to reduce logical I/O and improve application scalability.
Before a new feature or module goes online, perform stress testing with production data.
Before a new feature or module goes online, perform regression testing with production data.
System resource algorithm
The following simplified model explains the relationship among system resources, SQL statement execution costs, and queries per second (QPS):
Condition: The application model is constant, which means the application is not modified.
avg_lgc_io: The average logical I/O required to execute each query.
total_lgc_io: The total logical I/O that the instance's CPU resources can process per unit of time.
Formula:
total_lgc_io = avg_lgc_io × QPS. This meansTotal CPU resources per unit of time = Average logical I/O per query × Number of queries per unit of time.
References
Resolve high CPU utilization on an ApsaraDB RDS for MySQL instance using the autonomy service
Applicable versions
ApsaraDB RDS for MySQL
ApsaraDB RDS for MariaDB