When you use ApsaraDB for Redis, you may encounter performance degradation, deteriorated user experience, and large-scale failures if you do not identify and handle large keys or hotkeys in a timely manner. This topic describes the causes of large keys and hotkeys, the issues that may be caused by large keys and hotkeys, and how to identify and optimize large keys and hotkeys in a timely manner.
Definitions of large key and hotkey
|large key||The size of a key and the number of members in the key determine whether the key is considered a large key. The following list provides some examples:|
|hotkey||The frequency at which a key is requested determines whether the key is considered a hotkey. The following list provides some examples:|
Issues caused by large keys and hotkeys
Causes of large keys and hotkeys
Large keys and hotkeys may occur due to a variety of reasons, such as incorrect use of ApsaraDB for Redis, insufficient workload planning, accumulation of invalid data, and traffic spikes.
- Large keys
- Incorrect use of ApsaraDB for Redis: If ApsaraDB for Redis is used in an improper scenario, the size of a key may be larger than necessary. For example, if a STRING key is used to store a binary file that is large in size, the size of the key may be larger than necessary.
- Insufficient workload planning: Before a feature is released, no sufficient workload planning is conducted. For example, members are not properly split between keys and some keys have more members than required.
- Accumulation of invalid data: Invalid data is not deleted on a regular basis. For example, the number of members for a HASH key constantly increases because invalid data is not timely cleared.
- Code failures: Code failures occur on consumer applications that use LIST keys, which causes the members in the key to only increase.
- Unexpected traffic spikes: Unexpected traffic spikes may occur for a variety of reasons, such as viral products and news, a large number of "likes" flooding in from the viewers of a livestream, and a battle between multiple large teams in a game.
Identify large keys and hotkeys
ApsaraDB for Redis provides a variety of methods for you to identify large keys and hotkeys.
|Method||Benefit and drawback||Description|
|Use the real-time key statistics feature (recommended)||You can use the real-time key statistics feature to display the statistics of large keys and hotkeys in an instance in real time. You can also query the historical statistics of large keys and hotkeys that were generated within the last four days. You can use this feature to obtain key statistics such as the memory usage and access frequency. Then, you can troubleshoot issues and optimize instances based on the statistics.|
|Use the offline key analysis feature||The offline key analysis feature allows you to analyze RDB backup files of ApsaraDB for Redis instances in a customized manner and identify large keys in these instances. You can view the statistics of keys in an instance, such as the memory usage, distribution, and time-to-live (TTL) of keys. You can use these statistics to optimize the instance and prevent issues such as insufficient memory and performance degradation that are caused by improper distribution of keys.|
|Identify large keys and hotkeys by using the bigkeys and hotkeys parameters in redis-cli.|
The bigkeys parameter provided by ApsaraDB for Redis allows redis-cli to traverse all keys in an ApsaraDB for Redis instance and return the overall statistics of keys and the largest keys of each data type. The bigkeys parameter can return statistics for keys of six data types: STRING, LIST, HASH, SET, ZSET, and STREAM. Sample command:
Note If you want to analyze only large keys of the STRING type or identify the HASH keys that have more than 10 members, the bigkeys parameter cannot fulfill your needs.
As of Redis 4.0, the hotkeys parameter is provided to help you quickly identify hotkeys. For more information, see Query hotkeys in Redis 4.0.
|Analyze a specified key by using built-in commands of ApsaraDB for Redis||The following list provides low-risk commands for analyzing keys of various data types to determine whether a key is a large key: |
Note The DEBUG OBJECT and MEMORY USAGE commands consume large amounts of resources when they are run. Moreover, the time complexity of these commands is O(N), which indicates that these commands may block ApsaraDB for Redis instances. Therefore, we recommend that you do not use these commands.
|Identify hotkeys at the business layer||This method allows you to add code to the business layer to record requests that were sent to ApsaraDB for Redis instances and asynchronously analyze the collected statistics.|
|Identify large keys in a customized manner by using the redis-rdb-tools project||The redis-rdb-tools project is written in the Python programming language. redis-rdb-tools is an open source tool that can be used to analyze RDB files in a customized manner. You can analyze the memory usage of all keys in an ApsaraDB for Redis instance, and query and analyze statistics of each key in a fine-grained manner.|
|Identify hotkeys by using the MONITOR command||The MONITOR command that is available in ApsaraDB for Redis can display the statistics of all requests related to an instance, including statistics about time, clients, commands, and keys. |
In case of an emergency, you can run the MONITOR command and export the output to a file. You can then analyze and classify the requests in the output to identify hotkeys generated during the emergency period after you disable the MONITOR command.
Note However, the MONITOR command significantly degrades the performance of ApsaraDB for Redis instances. We recommend that you use the MONITOR command only in special cases.
Optimize large keys and hotkeys