If an ApsaraDB for Redis instance experiences high CPU utilization, the throughput of the instance and response time of an application that connects to the instance are affected. In extreme cases, the application may stop responding. If the average CPU utilization is higher than 50% and the average peak CPU utilization within a 5-minute period is higher than 90%, the stability of the application may be affected. You must pay close attention to and troubleshoot this issue.
Search for and disable commands that cause high CPU utilization
Commands that consume a large amount of CPU resources have a time complexity of O(N) or higher. In most cases, a command with a higher time complexity consumes more CPU resources. This increases CPU utilization. For more information about the time complexity of each command, visit the Redis official website.
If ApsaraDB for Redis runs commands that consume a large amount of CPU resources, pending requests are piled up in the queue due to single-threading. This slows down the response of applications. In some cases, an ApsaraDB for Redis instance may be overwhelmed by pending requests. An application may be disconnected due to these requests timing out. In addition, user requests may be directly forwarded to the backend database. As a result, a cache avalanche occurs.
- Use the performance monitoring feature to identify the time period during which CPU utilization is high. For more information, see Query monitoring data.
- Use the following methods to identify the commands that consume a large amount of
- Audit logs record modification and deletion operations that are performed on ApsaraDB for Redis instances. You can query audit logs to analyze the commands and trends within a specified time period. This allows you to identify the commands that consume a large amount of CPU resources. For more information, see Query audit logs.
- Slow logs record commands that are run for longer than the specified threshold. You
can identify commands that consume a large amount of CPU resources based on the statements
and durations that are recorded in slow logs. For more information, see Query slow logs.
Note The amount of time that is taken to execute a statement is measured in microseconds.
- Assess and disable commands that cause a high risk and consume a large amount of CPU resources, such as FLUSHALL, KEYS, and HGETALL. For more information, see Disable high-risk commands.
- Optimize your application. For example, do not frequently sort data.
- Optional:Use one of the following methods to modify the instance based on your business settings:
- Enable the proxy query cache feature. After you enable this feature, proxy servers cache the request and response data of hotkeys. If a proxy server receives a duplicate request during the validity period of the cached data, the proxy server directly returns a response to the client without the need to interact with backend data shards. This helps prevent skewed requests caused by hotkeys that receive a large number of read requests. For more information, see Use proxy query cache to address issues caused by hotkeys.
- Analyze the slow logs and audit logs, and then check the hotkeys on each node. This way, you can resolve the issue or slightly decrease CPU utilization. For more information, see Query real-time and historical hotkeys.
Optimize short-lived connections
Connections to an ApsaraDB for Redis instance are frequently established. As a result, a large amount of resources of the ApsaraDB for Redis instance are consumed. In this case, CPU utilization is high, the number of established connections is large, and the queries per second (QPS) does not reach the expected value.
Disable AOF persistence
By default, append-only file (AOF) persistence is enabled for ApsaraDB for Redis instances. If an ApsaraDB for Redis instance runs with heavy loads, frequent AOF operations may increase CPU utilization.
Disable AOF persistence if this does not adversely affect your business. In addition, you can back up the Redis data during off-peak hours or during the maintenance window to minimize the impact.
Optimize proxy server connections and the use of pipelines
The performance trends of a cluster instance or read/write splitting instance are displayed in the ApsaraDB for Redis console. The CPU utilization of proxy servers is unevenly distributed, and large differences exist between the maximum and minimum CPU utilization.
- Check whether connection usage is evenly distributed by using the performance trends feature. For more information, see Performance trends.
- Perform the following operations based on whether connection usage is evenly distributed:
- Evenly distributed: Restart the client or proxy server where business applications are deployed to redistribute connections.
- Unevenly distributed: Because uneven distribution is usually caused by a large scale of pipeline or batch operations, decrease the scale of corresponding operations. For example, you can separate one operation into multiple operations.
Evaluate the service performance
The preceding methods are used to optimize the performance of your instance. If the average CPU utilization still exceeds 50% during normal business operations, the instance may have a performance bottleneck.
To resolve this issue, first check for commands and requests from application hosts that may degrade the instance performance. If such commands or requests exist, you must optimize your business system. If no such commands or requests are found but the CPU utilization is still high, we recommend that you upgrade the instance specifications to ensure business stability. You can also upgrade the instance to a cluster instance or read/write splitting instance. For more information about how to upgrade an instance, see Change the configurations of an instance.