Tair (Redis® OSS-Compatible):Troubleshoot high CPU utilization

Causes of high CPU utilization

• CPU-intensive command: These commands, such as KEYS, HGETALL, or using MGET, MSET, HMSET, and HMGET to operate on many keys at once, have a time complexity of O(N) where N is a large value. Commands with higher time complexity consume more CPU resources, leading to increased CPU utilization.

  Because commands are executed on a single thread, running these commands can block the instance, causing other requests to queue and increasing application latency. In extreme cases, this can lead to timeouts and cause a cache avalanche, where traffic bypasses the cache layer and hits your backend database directly.

  Note

  For more information about the time complexity of each command, see Commands.

• Hotkey: When a small number of keys receive a disproportionately high number of requests, they become hotkeys. A hotkey can consume significant CPU resources and increase latency for other operations. In a cluster architecture, if hotkeys are concentrated on a few data nodes, it can cause CPU utilization skew, where the CPU utilization of those nodes becomes much higher than that of other nodes.

• Large key: Accessing a large key consumes more memory, CPU resources, and network bandwidth. A large key is more likely to become a hotkey, contributing to high CPU utilization. If large keys are concentrated on specific data nodes, it can lead to CPU utilization skew, as well as memory and bandwidth usage skew.

• Short-lived connection: Frequent connection creation and termination consumes significant CPU resources on connection handling rather than data processing.

• AOF: Append-only file (AOF) persistence is enabled by default. When an instance is under heavy load, disk I/O from the AOF process can increase CPU utilization and command latency.

Scenarios of high CPU utilization

High CPU utilization typically manifests in three common scenarios:

• During a specific time period, CPU utilization suddenly spikes, sometimes reaching 100%. For causes and solutions, see Sudden spike in CPU utilization.

• A specific data node has high CPU utilization while other data nodes have low utilization. For causes and solutions, see CPU utilization skew across data nodes.

• A specific proxy node has high CPU utilization while other proxy nodes have low utilization. For causes and solutions, see CPU utilization skew across proxy nodes.

Take the appropriate steps to reduce CPU utilization based on your scenario.

Sudden spike in CPU utilization

If the overall CPU utilization of your instance spikes, follow these steps to investigate and resolve the issue.

Identify and disable CPU-intensive commands

Procedure

1. Use the performance monitoring feature to identify the specific time range when CPU utilization was high. For more information, see View performance monitoring data.

2. Use the following tools to identify CPU-intensive commands:

   • The latency insights feature records the latency of all commands and custom events. You can use it to find commands with high latency during a specific time period on a specific node.

   • The slow query log feature records commands that exceed a specified execution time (20 ms by default). You can use it to find long-running, CPU-intensive commands.

Solutions

• Evaluate and disable high-risk and CPU-intensive commands, such as FLUSHALL, KEYS, and HGETALL. For more information, see Disable high-risk commands.

• Optional: Based on your business needs, consider the following instance adjustments:

  • Change the instance to a read/write splitting architecture to offload CPU-intensive commands or applications.

  • Change the instance to a DRAM-based instance to leverage its multi-threaded capabilities and reduce CPU utilization.

  Note

  For instructions on how to change the architecture and type of an instance, see Change the configurations of an instance.

Optimize for short-lived connections

Procedure

1. Use the performance monitoring feature to identify the specific time range when CPU utilization was high. For more information, see View performance monitoring data.

2. On the performance monitoring page, check for a combination of high CPU utilization, a high connection count, and a lower-than-expected queries per second (QPS). This pattern indicates an issue with short-lived connections.

Solutions

• Switch from short-lived connections to persistent connections, for example, by using a connection pool like JedisPool. For more information, see Use a client to connect to an instance.

• Change the instance to a DRAM-based instance, which is optimized for short-lived connections.

Disable AOF

AOF is enabled by default. Under heavy load, frequent AOF operations can contribute to high CPU utilization.

If your business requirements allow, consider disabling persistence and scheduling data backups during off-peak hours to minimize impact.

Evaluate service capacity

If the average CPU utilization remains high (over 70%) during normal operations after performing the preceding optimizations, your instance likely has a performance bottleneck.

First, check for abnormal access patterns, such as unusual commands or a high volume of requests from a specific application host. These issues should be addressed at the application level. If all access is legitimate, the high load is a normal result of your workload. To ensure stable operation, we recommend that you upgrade your instance specifications or change to a cluster architecture or read/write splitting architecture. For more information, see Change the configurations of an instance.

Data node CPU utilization skew

If you are using a cluster architecture or read/write splitting architecture, you may observe that some data nodes have high CPU utilization while others have low utilization. Follow these steps to investigate and resolve the issue.

Troubleshoot and optimize hotkeys

Procedure

1. Use the performance monitoring feature to identify the specific time range when CPU utilization was high. For more information, see View performance monitoring data.

2. On the history page of Real-time Key Statistics, select the data node with high CPU utilization, specify the time range from Step 1, and then click Search. This shows the hotkeys during the period of high CPU utilization.

Solutions

• Split the hotkey based on your business logic. For example, add a user ID or a timestamp range to the key name.

• If a high volume of read requests causes the hotkey, consider changing the instance to a read/write splitting architecture to reduce read pressure on each data node.

  Note

  In scenarios with extremely high request volumes, a read/write splitting architecture can introduce unavoidable replication latency, which may lead to reading stale data. Therefore, this architecture may not be the optimal solution for workloads with high read and write pressure that also require strong data consistency.

Identify and disable CPU-intensive commands

Procedure

1. Use the performance monitoring feature to identify the specific time range when CPU utilization was high. For more information, see View performance monitoring data.

2. Use the following tools to identify CPU-intensive commands:

   • The latency insights feature records the latency of all commands and custom events. You can use it to find commands with high latency during a specific time period on a specific node.

   • The slow query log feature records commands that exceed a specified execution time (20 ms by default). You can use it to find long-running, CPU-intensive commands.

Solution

Evaluate and disable high-risk and CPU-intensive commands, such as FLUSHALL, KEYS, and HGETALL. For more information, see Disable high-risk commands.

Troubleshoot and optimize large keys

Procedure

1. Use the performance monitoring feature to identify the specific time range when CPU utilization was high. For more information, see View performance monitoring data.

2. On the offline key analysis page, click Analyze Now. Select the data node with high CPU utilization and click OK. This shows the large keys that existed during the period of high CPU utilization.

Solution

Based on your business requirements, split large keys into smaller keys to distribute the request load.

Proxy node CPU utilization skew

If you are using a cluster architecture or a read/write splitting architecture, you may find that some proxy nodes have high CPU utilization while others have low utilization. Follow these steps to investigate and resolve the issue.

Procedure

On the Proxy Node tab of the Performance Trends page, check whether the connection utilization is balanced across nodes. For more information, see Performance Trends.

Solutions

Perform one of the following operations based on whether connection utilization is balanced:

• If connection utilization is balanced: Restart your client application or the proxy nodes to rebalance the connections. To restart a proxy node, see Restart or rebuild proxy nodes.

• If connection utilization is unbalanced: This imbalance is typically caused by large-scale pipeline or batch operations. Reduce the size of these operations by, for example, splitting a large batch operation into several smaller ones.