:Key expiration and eviction policies

Overview

This topic describes Key expiration and eviction policies , and introduces some best practices for using Redis.

• Redis expired Key cleanup policy
• Algorithm used to delete expired keys
• Data eviction algorithm
• Precautions for using Redis

Detail

Redis expired Key cleanup policy

To prevent the apsaradb for Redis service from being affected when a large number of expired keys are cleared at a time, apsaradb for Redis clears expired keys when the CPU is idle. The specific Redis eviction policy for expired keys is as follows:

1. The device checks whether the Key has expired. If the Key has expired, the device evicts the expired Key.

     robj *lookupKeyRead(redisDb *db, robj *key) {     robj *val;     expireIfNeeded(db,key);     val = lookupKey(db,key);     ...     return val; }

2. When the CPU is idle, it uses the serverCron scheduled task to evict partial expired keys.
   

   aeCreateTimeEvent(server.el, 1, serverCron, NULL, NULL)  int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {     ...       databasesCron();     ... }  void databasesCron(void) {     /* Expire keys by random sampling. Not required for slaves      + as master will synthesize DELs for us. */     if (server.active_expire_enabled && server.masterhost == NULL)         activeExpireCycle(ACTIVE_EXPIRE_CYCLE_SLOW);     ... }

3. Each time the event loop is executed, some expired keys are evicted.

   void aeMain(aeEventLoop *eventLoop) {     eventLoop->stop = 0;     while (! eventLoop->stop) {         if (eventLoop->beforesleep ! = NULL)             eventLoop->beforesleep(eventLoop);         aeProcessEvents(eventLoop, AE_ALL_EVENTS);      } }
   
   void beforeSleep(struct aeEventLoop *eventLoop) {
       ...
       /* Run a fast expire cycle (the called function will return
        - ASAP if a fast cycle is not needed). */
       if (server.active_expire_enabled && server.masterhost == NULL)
          activeExpireCycle(ACTIVE_EXPIRE_CYCLE_FAST);
       ...
   }

Algorithm used to delete expired keys

The mechanism of clearing expired keys limits the frequency and maximum time for clearing expired keys. In this way, expired keys can be cleared without affecting the normal service for a long time to achieve the optimal performance.

Apsaradb for Redis periodically and randomly tests and processes expired keys. When these keys are deleted, the algorithm is as follows:

• The Redis configuration items hz parameter specifies the frequency that serverCron scheduled tasks are executed. The default value is 10, indicating that tasks are executed every second when the CPU is idle.
• The clearance time of each expired Key does not exceed 25% of the CPU time. That is, if hz is equal to 1, the maximum clearance time is 250ms. If hz is equal to 10, the maximum clearance time is 25ms.
• Indicates that expired keys are cleared each time all the related databases are traversed in sequence.
• It randomly selects 20 keys from one database and determines whether to expire. If the Key expires, the expired keys are evicted.
• If more than five keys have expired, repeat the previous step. Otherwise, traverse the Next Database.
• If the disk usage reaches 25% of the CPU time during the cleanup, the disk exits the cleanup process.

This is a simple probability-based algorithm. Assuming that the samples are almost representative of the entire Key, apsaradb for Redis deletes expired keys until the percentage of expired keys falls below 25%. In the long run, the maximum number of expired keys that still occupy memory space at any given time is the number of write operations per second divided by 4. The specific considerations are as follows:

• The algorithm uses a random Key acquisition method to determine whether the Key has expired. Therefore, it is almost impossible to clear all the expired keys.
• You can increase the hz parameter to increase the cleanup frequency so that expired keys can be processed in a more timely manner. However, a high hz value increases CPU time consumption, which should be configured based on your actual business needs.

Data eviction algorithm

Based on the eviction policy set by the user, select the keys to be evicted until the current memory is less than the maximum memory. For more information about the eviction policy, see the default data eviction policy.

Precautions for using Redis

• Do not store junk data and delete junk data in a timely manner.
  That is, experimental data and offline business data need to be deleted in a timely manner.
• Set the expiration time for each Key.
  Apsaradb for Redis allows you to set an expiration time for keys that have a validity period. You can use the expired Key cleanup policy of apsaradb for Redis to delete expired keys. This reduces the memory usage of expired keys, and reduces the O&M costs. You do not need to delete expired keys on a regular basis.
• The size of a single Key must not be too large.
  The Redis single string value is 43 MB, or the list has millions of members, taking up 1 GB of memory. When accessing such a Key, the network transmission latency is high, the required output buffer is also large, and it is also likely to cause relatively high latency when regular cleanup. Therefore, it is best to use business splitting or data compression to avoid such excessive keys.
• If the source instance and the destination instance use the same Redis database for different business, we recommend that you use different logical reservoir instances to divide the business.
  Both the expired Key clearing policies and the expiration policies of apsaradb for Redis traverse all databases. Therefore, distributing expired keys in different databases can help you clean up expired keys in a timely manner. In addition, when different databases are used for different services, it is helpful for troubleshooting and timely deprecation of useless data.

Documentation

For more information, see Key expiration and eviction policies.

Application scope

• ApsaraDB for Redis