Tair:Identify and handle large keys and hotkeys

Term

Description

large key

The size and number of members determine whether a key is considered large. The following section lists some examples:

• A key of a large size is considered a large key. Example: a string key that is 5 MB in size.

• A key that has a large number of members is considered a large key. Example: a ZSET key that has 10,000 members.

• A key whose member data is large in size is considered a large key. Example: a hash key that has only 1,000 members whose total size is 100 MB.

hotkey

The frequency at which a key is requested determines whether the key is a hotkey. The following section lists some examples:

• A key that receives a large number of queries per second (QPS) is considered a hotkey. For example, if a Tair instance has a total QPS of 10,000 and one key in the instance receives 7,000 QPS, the key is considered a hotkey.

• A key that has a high bandwidth usage is considered a hotkey. For example, if a hash key that has thousands of members and is 1 MB in size receives a large number of HGETALL commands per second, the key is considered a hotkey.

• A key that has a high CPU utilization is considered a hotkey. For example, if a ZSET key that has tens of thousands of members receives a large number of ZRANGE commands per second, the key is considered a hotkey.

Category

Description

Large key

• It takes longer for the client to run commands.

• Operations may be blocked, important keys may be evicted, or out-of-memory (OOM) errors may even occur when the memory usage of a Tair instance reaches the upper limit specified by the maxmemory parameter.

• The memory usage of a data shard in a Tair cluster instance exceeds that of other data shards, which results in imbalanced memory usage across data shards in the instance.

• When a read request is made for a large key, the response time may increase and other services may be affected. This is because the bandwidth of the Tair instance to which the key belongs is exhausted.

• The primary database may be blocked for an extended period of time while a large key is being deleted. This may lead to a synchronization failure or a master-replica switchover.

Hotkey

• Hotkeys cause high CPU usage, which slows down the response time of other requests and degrades the performance of Tair instances.

• Request skews may take place for Tair cluster instances. Request skews occur when one data shard in an instance receives a large number of requests while other data shards in the instance remain idle. During a request skew, the maximum number of connections to a data shard may be reached and new connections to the shard may be rejected.

• In flash sales, overselling occurs if the key corresponding to a commodity receives more requests than can be handled by Tair.

• A cache breakdown occurs if a hotkey receives more requests than can be handled by Tair. In this case, a large number of requests are directly sent to backend storage and may lead to a storage breakdown. This affects other business.

Method

Benefit and drawback

Description

Use the real-time key statistics feature (recommended)

• Benefits: This method has high precision and minimal impacts on performance.

• Drawbacks: The number of keys displayed is limited, but sufficient for most common scenarios.

You can use the key analysis feature to display the statistics of large keys and hotkeys in an instance in real time. You can also query the historical statistics of large keys and hotkeys within the last four days. You can use this feature to obtain key statistics such as the memory usage and access frequency. Then, you can troubleshoot issues and optimize instances based on the statistics.

Use the offline key analysis feature

• Benefits: This method allows you to analyze historical backup files without affecting online services.

• Drawbacks: This method has poor timeliness and takes a large amount of time to analyze large Redis database (RDB) files.

The cache analysis feature allows you to analyze RDB backup files of Tair instances in a customized manner and identify large keys in these instances. You can view the statistics of keys in an instance, such as the memory usage, distribution, and expiration time of keys. You can use these statistics to optimize the instance and prevent issues such as insufficient memory and performance degradation that are caused by improper distribution of keys.

Identify large keys and hotkeys by using the bigkeys and hotkeys parameters in redis-cli.

• Benefits: This method is convenient, fast, and secure.

• Drawbacks: This method does not support customized analysis and provides limited precision and poor timeliness.

As of Redis 4.0, the hotkeys parameter is provided to help you quickly identify hotkeys. Sample command: redis-cli -h r-***************.redis.rds.aliyuncs.com -a <password> --hotkeys.

Analyze a specified key by using built-in commands of open source Redis

• Benefits: This method is convenient and has little impact on online services.

• Drawbacks: The returned serialized length of a key is not equal to the actual length of the key in memory. This method has limited precision and is for reference only.

The following section lists low-risk commands for analyzing keys of various data types to determine whether a key is a large key.

• For a string key, you can run the STRLEN command. This command returns the length (number of bytes) of a string value stored in the key.

• For a list key, you can run the LLEN command. This command returns the length of a list value stored in the key.

• For a hash key, you can run the HLEN command. This command returns the number of members in the key.

• For a set key, you can run the SCARD command. This command returns the number of members in the key.

• For a ZSET key, you can run the ZCARD command. This command returns the number of members in the key.

• For a stream key, you can run the XLEN command. This command returns the number of members in the key.

Identify hotkeys at the business layer

• Benefits: This method can identify hotkeys in a timely manner with high precision.

• Drawbacks: To implement this method, you must write more complex business code. Moreover, this method may degrade performance.

This method allows you to add code to the business layer to record requests that were sent to Tair instances and asynchronously analyze the collected statistics.

Identify large keys in a customized manner by using the redis-rdb-tools project

• Benefits: This method supports customized analysis without affecting online services.

• Drawbacks: This method has poor timeliness and takes a large amount of time to analyze large RDB files.

The redis-rdb-tools project is written in the Python programming language. redis-rdb-tools is an open source tool that can be used to analyze RDB files in Tair in a customized manner. You can analyze the memory usage of all keys in a Tair instance, and query and analyze statistics of the keys in a fine-grained manner.

Identify hotkeys by using the MONITOR command

• Benefits: This method is convenient and secure.

• Drawbacks: This method consumes CPU, memory, and network resources and has poor timeliness and limited precision.

The MONITOR command that is available in Tair can display the statistics of all requests related to an instance, including statistics about time, clients, commands, and keys.

In case of an emergency, you can run the MONITOR command and export the output into a file. You can then analyze and classify the requests in the output to identify hotkeys generated during the emergency period after you disable the MONITOR command.

Category

Solution

Large key

• Split large keys

  For example, you can split a hash key that contains tens of thousands of members into multiple hash keys that have proper numbers of members. For Tair cluster instances, you can split large keys to balance the memory usage across multiple data shards.

• Delete large keys

  You can store data that is not suitable for Tair to process in other devices and delete the data from Tair.

  Note

  You can run the UNLINK command to safely delete large keys or super large keys. This command can be used to gradually delete keys from Tair to prevent it from being blocked.

• Monitor the memory usage of Tair

  You can specify proper memory usage thresholds in the monitoring system for Tair to send alerts. You can specify 70% as the memory usage threshold and 20% as the threshold of memory usage increase over a one-hour period. This way, you can prevent potential problems. For example, you can send alerts in advance to prevent an increase in the number of keys caused by the failure of a consumer application that uses list data. For more information, see Alerting settings.

• Delete expired data on a regular basis

  The accumulation of expired data leads to large keys. For example, if you incrementally write a large amount of data to hash keys and ignore the timeliness of the data, large keys may be generated. You can use scheduled tasks to delete invalid data.

  Note

  To prevent Tair from being blocked when you delete invalid hash data, we recommend that you run the HSCAN and HDEL commands.

• Use Tair

  If you have a large number of hash keys and want to delete a large number of invalid members from some keys, scheduled tasks are insufficient to delete invalid members in a timely manner. In this case, you can use Tair.

  Tair provides the TairHash data structure. TairHash is a hash data type that allows you to specify expiration time and version numbers for fields. Similar to Redis HASH, TairHash provides a variety of data interfaces and high processing performance. However, Redis HASH allows only expiration time to be specified for keys. TairHash is more flexible and simplifies application development in most scenarios. In addition, TairHash uses the active expiration algorithm to check the expiration time of fields and delete expired fields without increasing the database response time.

  The preceding advanced features can be used to improve O&M efficiency, reduce troubleshooting workloads, and simplify business code. For more information, see exHash.

Hotkey

• Replicate hotkeys for Tair cluster instances

  Requests made for a hotkey in a data shard cannot be redistributed to other data shards because the minimum granularity of hotkey migration in a Tair cluster instance is key. This results in a constant high workload for the single data shard. In this case, you can replicate the hotkey in the data shard to generate new hotkeys and migrate the new hotkeys to other data shards. For example, you can replicate a hotkey named foo in a data shard to generate three identical hotkeys named foo2, foo3, and foo4. Then, you can migrate foo2, foo3, and foo4 to other data shards to reduce the pressure on the single data shard.

  Note

  The drawback of this method is that you must modify the corresponding code and data inconsistency may occur because you must update multiple keys instead of one key. For this reason, we recommend that you use only this solution as a temporary one.

• Use a read/write splitting architecture

  If the accumulation of read requests causes hotkeys, you can change your instance into a read/write splitting instance to reduce the read pressure imposed on each data shard of the instance, or increase the number of replica nodes for the instance. However, the read/write splitting architecture increases the complexity of both the business code and the instance. You must provide server load balancing tools such as proxies and Linux Virtual Server (LVS) for multiple replica nodes and prepare to deal with the increased failure rate caused by a significant increase in the number of replica nodes. If you change your instance into a Tair cluster instance, you may encounter bigger challenges in monitoring, O&M, and troubleshooting.

  In response to these challenges, Tair provides out-of-the-box solutions. You can modify your instance architecture as your needs evolve by making a configuration change, such as changing a master-replica instance into a read/write splitting instance or a read/write splitting instance into a cluster instance. For more information, see Change the configurations of an instance.

  Note

  The read/write splitting architecture also has its drawbacks. If a large number of requests are sent, inevitable latency exists for read/write splitting instances and dirty data may be read from these instances. Therefore, the read/write splitting architecture is not optimal for scenarios that have high requirements for read and write capabilities and data consistency.

• Use the proxy query cache feature of Tair

  Tair uses effective sorting and statistical algorithms to identify hotkeys that receive more than 3,000 queries per second (QPS). After you enable the proxy query cache feature, proxy nodes cache request and response data of hotkeys based on the rules you set. Proxy nodes cache only request and response data of a hotkey, instead of the entire key. If a proxy node receives a duplicate request within the validity period of the cached data, the proxy server directly returns the response of the request to the client without the need to interact with backend data shards. This improves the read speed, reduces the impacts of hotkeys on the performance of data shards, and prevents skewed requests.

  After this feature is enabled for a Tair instance, duplicate requests from clients are directly sent to proxy nodes instead of backend data shards. The proxy nodes then return responses to the clients. Requests made for hotkeys can be processed by multiple proxy nodes instead of a single data node, which significantly reduces the hotkey workload on data nodes. The proxy query cache feature of Tair also provides a variety of commands for you to query and manage proxy query cache. For example, you can run the querycache keys command to query all cached hotkeys and run the querycache listall command to query all cached commands. For more information, see Use proxy query cache to address issues caused by hotkeys.