As one of the most popular NoSQL databases, Redis earned its name by offering outstanding performance and extensive data structure support. Besides, it never stops evolving and, ever since its birth, it has been continuously launching new features.
Redis 4.0, the latest GA version in the community, has been released for almost a year, and ApsaraDB for Redis 4.0 has been released for half a year. There are a series of articles that analyze Redis 4.0's new features from the perspective of source code implementation. This article will discuss how to quickly understand and use benefits of Redis 4.0 from the perspective of users.
Many of you must have encountered the problem of deleting large keys. In addition to strings, Redis also supports list, set, hash, sorted set, and other complex data structures. These data structures have enriched Redis usage. However, improper usage may cause problems of over-sized single keys.
To better understand the problem, let's look at a simple example. Let's assume that there is an influential social networking account, "Mr. V", who has millions of subscribers. We can use the set type data structure to store Mr. V's fans' IDs. The key that stores the fans set is called funs; let's take a look at the number of the subscribers:
127.0.0.1:6379> SCARD funs (integer) 6320505
Mr. V has over 6 million fans! Unfortunately, Mr. V canceled his account one day and we need to delete his information. Let's use the DEL command to delete this key.
127.0.0.1:6379> DEL funs (integer) 1 (3.11s) 127.0.0.1:6379> slowlog get 1) 1) (integer) 4 2) (integer) 1528169923 3) (integer) 3104812 4) 1) "DEL" 2) "funs" 5) "127.0.0.1:48398" 6) ""
Redis 4.0 expands slowlog's return results and displays the IP:Port of clients that generated slow logs for convenient tracing.
We can see that the deletion operation took 3 seconds. This means that Redis was unable to execute other commands within these 3 seconds. This may cause adverse effect for online businesses. How can we avoid the blockage caused by deleting large keys? Redis 4.0 launched the Lazyfree function. Lazyfree uses the UNLINK command to delete large keys. The main thread is only responsible for removing the key from the database and the actual release operation is done by BIO backend threads. Let's take a look at the result:
127.0.0.1:6379> UNLINK funs (integer) 1 (3.11s) 127.0.0.1:6379> slowlog get (empty list or set)
We can see that the UNLINK execution was very fast and no slowlog was generated.
Lazyfree involves three commands:
And 4 configuration items:
Redis is built with a Lua environment to allow users to expand features. However, for the sake of consistency, the scripts must be pure functions. In other words, when a Lua script is given the same parameters, the same results will be returned for repeated executions.
Why? Because Redis not only is a standalone memory database, Redis also supports Master/slave replication and persistence. An executed Lua script will be replicated to the slave and be persisted to disk. If the results for repeated executions are not the same, then data in memory, disk, and slave will be inconsistent. This will cause data disorder after failover and restart and cause adverse effect to the business.
Here is an actual example. Assume that we have a Lua script to simply record the current time:
local now = redis.call('time') redis.call('set','now',now) return redis.call('get','now')
It uses Redis' TIME command to obtain the timestamp, and store the timestamp to a key named now. However, an exception was thrown when executing this script:
$redis-cli --eval escript (error) ERR Error running script (call to f_cfba5ec6a699dad183456f19d1099d8dabfdb80c): @user_script:3: @user_script: 3: Write commands not allowed after non deterministic commands. Call redis.replicate_commands() at the start of your script in order to switch to single commands replication mode.
The error message is simple: Write commands not allowed after non deterministic commands. Because TIME is random, Redis does not allow executing the write command, in order to ensure data consistency. How can we achieve the random write operation? The answer is provided in the above error message:
call redis.replicate_commands(). After executing
redis.replicate_commands(), Redis does not synchronize the complete Lua script to Slave or persist it to disk. Instead, it directly replicates the write command to call Redis from the script and then the same results will be returned at the slave and after persistence.
The script is modified as follows:
redis.replicate_commands() local now = redis.call('time') redis.call('set','now',now) return redis.call('get','now')
Execute it again to achieve random write:
$redis-cli --eval escript "1528191578" $redis-cli --eval escript "1528191804"
LFU is a memory eviction policy newly introduced in Redis 4.0. LFU provides a more precise memory eviction algorithm. Its core function is to record the key access frequency within a period of time. It brings an additional benefit - hotspot key discovery.
Simply put, LFU uses 0-255 to indicate the access frequency of a key. A larger value indicates a higher frequency. The frequency increase is recorded in a logarithm-based manner. When LFU is 255, it indicates the key has been accessed by 1 million times.
We can use the OBJECT FREQ command to obtain a specified key's access frequency. But before that, we must set the memory eviction policy to allkeys-lfu or volatile-lfu:
127.0.0.1:6379> config get maxmemory-policy 1) "maxmemory-policy" 2) "noeviction" 127.0.0.1:6379> object freq counter:000000006889 (error) ERR An LFU maxmemory policy is not selected, access frequency not tracked. Please note that when switching between policies at runtime LRU and LFU data will take some time to adjust. 127.0.0.1:6379> config set maxmemory-policy allkeys-lfu OK 127.0.0.1:6379> object freq counter:000000006889 (integer) 3
Use the SCAN command to traverse all keys, use OBJECT FREQ to retrieve the access frequency, and then sort the results to obtain the hotspot key. To ease user usage, Redis' built-in client redis-cli also provides the hotspot discovery function. You can simply append
--hotkeys to the end of
redis-cli and then execute it:
$./redis-cli --hotkeys # Scanning the entire keyspace to find hot keys as well as # average sizes per key type. You can use -i 0.1 to sleep 0.1 sec # per 100 SCAN commands (not usually needed). [00.00%] Hot key 'counter:000000000002' found so far with counter 87 [00.00%] Hot key 'key:000000000001' found so far with counter 254 [00.00%] Hot key 'mylist' found so far with counter 107 [00.00%] Hot key 'key:000000000000' found so far with counter 254 [45.45%] Hot key 'counter:000000000001' found so far with counter 87 [45.45%] Hot key 'key:000000000002' found so far with counter 254 [45.45%] Hot key 'myset' found so far with counter 64 [45.45%] Hot key 'counter:000000000000' found so far with counter 93 -------- summary ------- Sampled 22 keys in the keyspace! hot key found with counter: 254 keyname: key:000000000001 hot key found with counter: 254 keyname: key:000000000000 hot key found with counter: 254 keyname: key:000000000002 hot key found with counter: 107 keyname: mylist hot key found with counter: 93 keyname: counter:000000000000 hot key found with counter: 87 keyname: counter:000000000002 hot key found with counter: 87 keyname: counter:000000000001 hot key found with counter: 64 keyname: myset
Memory analysis helps optimize Redis usage. The completely new MEMORY command can help users perform this operation. The MEMORY command has 5 sub-commands. These 5 sub-commands can be viewed from MEMORY HELP.
127.0.0.1:6379> memory help 1) "MEMORY DOCTOR - Outputs memory problems report" 2) "MEMORY USAGE <key> [SAMPLES <count>] - Estimate memory usage of key" 3) "MEMORY STATS - Show memory usage details" 4) "MEMORY PURGE - Ask the allocator to release memory" 5) "MEMORY MALLOC-STATS - Show allocator internal stats"
You can learn more about theses sub-commands on Redis's official documentation.
We looked at some of the key features of Redis 4.0, focusing on features like Lazytree, Lua, and Memory. To learn more about Redis on Alibaba Cloud, visit the ApsaraDB for Redis product page.
Alibaba Cloud MaxCompute - June 22, 2020
ApsaraDB - October 12, 2020
Alibaba Cloud Security - January 16, 2020
ApsaraDB - July 11, 2018
Alibaba Clouder - April 18, 2018
kirito.moe - April 14, 2020
A key value database service that offers in-memory caching and high-speed access to applications hosted on the cloudLearn More
An on-demand database hosting service for MySQL, SQL Server and PostgreSQL with automated monitoring, backup and disaster recovery capabilitiesLearn More
An on-demand database hosting service for PostgreSQL with automated monitoring, backup and disaster recovery capabilitiesLearn More
An on-demand database hosting service for MySQL with automated monitoring, backup and disaster recovery capabilitiesLearn More
More Posts by ApsaraDB