When you use Tair, if you do not identify and handle large keys or hotkeys in a timely manner, you may encounter performance degradation, deteriorated user experience, and large-scale failures. This topic describes the causes of large keys and hotkeys, issues that may be caused by large keys and hotkeys, and how to identify and optimize large keys and hotkeys in a timely manner.
Definitions of large key and hotkey
The size and number of members determine whether a key is considered large. The following section lists some examples:
The frequency at which a key is requested determines whether the key is a hotkey. The following section lists some examples:
The values used in the preceding examples are for reference only. You must determine whether a key is a large key or a hotkey based on your business requirements for Tair.
Issues caused by large keys and hotkeys
Causes for large keys and hotkeys
Large keys and hotkeys may occur for a variety of reasons, such as incorrect use of Tair, insufficient workload planning, accumulation of invalid data, and traffic spikes.
Incorrect use of Tair: If Tair is used in an improper scenario, the size of a key may be larger than necessary. For example, if a string key is used to store a binary file that is large in size, the size of the key may be larger than necessary.
Insufficient workload planning: Insufficient workload planning is conducted before a feature is released. For example, members are not split between keys properly and some keys have more members than required.
Accumulation of invalid data: Invalid data is not deleted on a regular basis. For example, the number of members for a hash key constantly increases because invalid data is not timely cleared.
Code failures: Code failures occur on a consumer business application that uses a list key. This allows members in the key to increase but not decrease.
Unexpected traffic spikes: Unexpected traffic spikes may occur for a variety of reasons, such as viral products and news, a large number of "likes" flooding in from the viewers of a livestream, or a battle between multiple large teams in a game.
Identify large keys and hotkeys
Tair provides a variety of methods for you to identify large keys and hotkeys.
Benefit and drawback
Use the real-time key statistics feature (recommended)
You can use the key analysis feature to display the statistics of large keys and hotkeys in an instance in real time. You can also query the historical statistics of large keys and hotkeys within the last four days. You can use this feature to obtain key statistics such as the memory usage and access frequency. Then, you can troubleshoot issues and optimize instances based on the statistics.
The cache analysis feature allows you to analyze RDB backup files of Tair instances in a customized manner and identify large keys in these instances. You can view the statistics of keys in an instance, such as the memory usage, distribution, and expiration time of keys. You can use these statistics to optimize the instance and prevent issues such as insufficient memory and performance degradation that are caused by improper distribution of keys.
Identify large keys and hotkeys by using the bigkeys and hotkeys parameters in redis-cli.
The bigkeys parameter provided by open source Redis allows redis-cli to traverse all keys in a Tair instance and return the statistics of keys and the largest keys of each data type. The bigkeys parameter can return statistics for keys of six data types: string, list, hash, set, ZSET, and stream. Sample command:
If you want to analyze only large string keys or identify the hash keys that have more than 10 members, the bigkeys parameter cannot fulfill your needs.
As of Redis 4.0, the hotkeys parameter is provided to help you quickly identify hotkeys. Sample command:
Analyze a specified key by using built-in commands of open source Redis
The following section lists low-risk commands for analyzing keys of various data types to determine whether a key is a large key.
The DEBUG OBJECT and MEMORY USAGE commands consume large amounts of resources when they are run. Moreover, the time complexity of these commands is O(N), which indicates that these commands may block Tair instances. We recommend that you do not use these commands.
Identify hotkeys at the business layer
This method allows you to add code to the business layer to record requests that were sent to Tair instances and asynchronously analyze the collected statistics.
Identify large keys in a customized manner by using the redis-rdb-tools project
The redis-rdb-tools project is written in the Python programming language. redis-rdb-tools is an open source tool that can be used to analyze RDB files in Tair in a customized manner. You can analyze the memory usage of all keys in a Tair instance, and query and analyze statistics of the keys in a fine-grained manner.
Identify hotkeys by using the MONITOR command
The MONITOR command that is available in Tair can display the statistics of all requests related to an instance, including statistics about time, clients, commands, and keys.
In case of an emergency, you can run the MONITOR command and export the output into a file. You can then analyze and classify the requests in the output to identify hotkeys generated during the emergency period after you disable the MONITOR command.
However, the MONITOR command significantly degrades the performance of Tair instances. We recommend that you use MONITOR commands only in special cases.
Optimize large keys and hotkeys