Redis+KVStore: Disk-based Storage for Massive Data

This article aims to address these questions by using the Ardb protocol, specifically at the encoding and decoding layer during the integration between Redis and KVStore.

Redis_and_Memcached

Abstract:What do we do when data exceeds the capacity but has to be stored on disks? How can we encapsulate KVStore and integrate it into Redis? How is Redis encoding implemented?

This article aims to address these questions by using the Ardb protocol, specifically at the encoding/decoding layer during the integration between Redis and KVStore.

Redis is currently a hot property in the NoSQL circle. It is multipurpose and practical, and especially suitable for cracking some challenges that fall beyond the capability of traditional relational databases. Redis, as a memory database, stores all the data in memory.

Ardb is a NoSQL storage service fully compatible with the Redis protocol. Its storage is implemented based on the existing mature KVStore engine. Theoretically, any KVStore implementations similar to B-Tree/LSM Tree can be used as Ardb's underlying storage. Ardb currently supports LevelDB/RocksDB/LMDB.

Encoding Mode

The encoding/decoding layer is a very important part of the Redis and KVStore integration solution. Through this layer, we can remove the differences between various KVStore implementations. You can encapsulate and implement complicated data structures in Redis such as string, hash, list, set, and sorted set with any simple KVStore engine.

For strings, it is clear that it can be mapped to a KV pair in a one-to-one manner in KVStore. For other container types, we need to do the following:

• One KV stores the metadata of the entire key (such as the number of members of the list and their expiration time).
• Each member needs a KV to save the member's name and value.

For sorted set, each member has two attributes: score and rank, so we need to do the following:

• One KV stores the metadata of the entire key.
• Each member needs a KV to store the score information.
• Each member needs a KV to store the rank information of each member.

Key Encoding Format

All keys contain the same prefix, and the encoding format is defined as follows:

[<namespace>] <key> <type> <element...>
The "namespace" is used to support something similar to database in Redis. It can be any string, and is not limited to a number.

The "key" is a varbinary string.
The "type" is used to define a simple key-value container. This type implicitly indicates the data structure type of the key. It is one byte long.

The key type in the "meta" information is fixed as KEY_META. Specific types will be defined in the value (refer to the next section).

In addition to the above three parts, different types of keys may have additional fields. For example, hash's key may need an additional "field" field.

Value Encoding Format

Internal values are complex, but their encoding all starts with "type", as defined in the previous section.

<type> <element...>

Subsequent formats vary based on various type definitions.

Encoding of various data types

The encoding of each type of data is shown as follows: "ns" stands for namespace.

            KeyObject                             ValueObject
String      [<ns>] <key> KEY_META                 KEY_STRING <MetaObject>
Hash        [<ns>] <key> KEY_META                 KEY_HASH <MetaObject>
            [<ns>] <key> KEY_HASH_FIELD <field>   KEY_HASH_FIELD <field-value>
Set         [<ns>] <key> KEY_META                 KEY_SET <MetaObject>
            [<ns>] <key> KEY_SET_MEMBER <member>  KEY_SET_MEMBER
List        [<ns>] <key> KEY_META                 KEY_LIST <MetaObject>
            [<ns>] <key> KEY_LIST_ELEMENT <index> KEY_LIST_ELEMENT <element-value>
Sorted Set  [<ns>] <key> KEY_META                 KEY_ZSET <MetaObject>
            [<ns>] <key> KEY_ZSET_SCORE <member>  KEY_ZSET_SCORE <score>
            [<ns>] <key> KEY_ZSET_SORT <score> <member> KEY_ZSET_SORT

ZSet encoding example

Here we use the most complex type, sorted set, as an example. Suppose that there is a Sorted Set A: {member = first, score = 1}, {member = second, score = 2}. Its storage mode in Ardb is as follows:

The storage encoding of Key A is:

// // The "|" in the pseudo-code represents the division of the domain and does not mean to store the data as "|". During actual serialization, each field is serialized at a specific location.
The key is: ns|1|A (1 stands for KEY_META metadata type). 
The value is: metadata encoding (redis data type/zset, expiration time, number of members, maximum and minimum score among others).

The core information storage encoding of Member "first" is:

The key is: ns|11|A|first (11 stands for the KEY_ZSET_SCORE type).
The value is: 11|1 (11 stands for the KEY_ZSET_SCORE type. 1 stands for the score of the Member "first").

The rank information storage encoding of Member "first" is:

The key is: ns|10|A|1|first (10 stands for the KEY_ZSET_SORT type and 1 stands for the score).
The value is: 10 (representing the KEY_ZSET_SORT type, insignificant.  RocksDB automatically sorts the values by key, so it is easy to calculate the rank, requiring no storage and updating).

The score information storage encoding of Member "second" is skipped.

When you use the zcard A command, you can access namespace_1_A directly to get the number of ordered collections in the metadata.

When you use zscore A first, you can directly access namespace_A_first to get the score of Member "first".

When you use zrank A first, first you run zscore to get the score, and then find the serial number of namespace_10_A_1_first.

The specific storage code is as follows:

KeyObject meta_key(ctx.ns, KEY_META, key);
ValueObject meta_value;
for (each_member) {
  // KEY_ZSET_SORT stores the rank information. 
  KeyObject zsort(ctx.ns, KEY_ZSET_SORT, key);
  zsort.SetZSetMember(str);
  zsort.SetZSetScore(score);
  ValueObject zsort_value;
  zsort_value.SetType(KEY_ZSET_SORT);
  GetDBWriter().Put(ctx, zsort, zsort_value); 

  // Store the score information. 
  KeyObject zscore(ctx.ns, KEY_ZSET_SCORE, key);
  zscore.SetZSetMember(str);
  ValueObject zscore_value;
  zscore_value.SetType(KEY_ZSET_SCORE);
  zscore_value.SetZSetScore(score);
  GetDBWriter().Put(ctx, zscore, zscore_value);
}
if (expiretime > 0)
{
    meta_value.SetTTL(expiretime);
}
// Metadata. 
GetDBWriter().Put(ctx, meta_key, meta_value);

Implementation of Del

All data structures store a key-value of the metadata using a uniform encoding format, so it is impossible to have the same name for different data structures. (That is why in the KV pairs storing the key, the K is fixed to the KEY_META type, and the corresponding type of information exists in the Value field of the Metadata type in Redis.)
When implementing Del, the system will first query the metadata key-value to get the specific data structure type, and then perform the corresponding deletion, following steps similar to the following:

• Query the meta information of the specified key to get the data structure type;
• Perform deletion according to the specific type;
• One "del" will require at least one read + subsequent write operation for the deletion.

The specific code is as follows:

int Ardb::DelKey(Context& ctx, const KeyObject& meta_key, Iterator*& iter)
{
  ValueObject meta_obj;
  if (0 == m_engine->Get(ctx, meta_key, meta_obj))
  {
    // Delete directly if the data is of the string type. 
    if (meta_obj.GetType() == KEY_STRING)
    {
      int err = RemoveKey(ctx, meta_key);
      return err == 0 ? 1 : 0;
    }
  }
  else
  {
    return 0;
  }

  if (NULL == iter)
  {
    // If the data is of a complicated type, the database will be traversed based on the namespace, key and type prefix. 
    // Search all the members with the prefix of  namespace|type|Key. 
    iter = m_engine->Find(ctx, meta_key);
  }
  else
  {
    iter->Jump(meta_key);
  }

  while (NULL != iter && iter->Valid())
  {
    KeyObject& k = iter->Key();
    ...
    iter->Del();
    iter->Next();
  }
}

The prefix search code is as follows:

Iterator* RocksDBEngine::Find(Context& ctx, const KeyObject& key) {
  ...
  opt.prefix_same_as_start = true;
  if (!ctx.flags.iterate_no_upperbound)
  {
    KeyObject& upperbound_key = iter->IterateUpperBoundKey();
    upperbound_key.SetNameSpace(key.GetNameSpace());
    if (key.GetType() == KEY_META)
    {
      upperbound_key.SetType(KEY_END);
    }
    else
    {
      upperbound_key.SetType(key.GetType() + 1);
    }
    upperbound_key.SetKey(key.GetKey());
    upperbound_key.CloneStringPart();
  }
  ...
}

Implementation of Expire

It is relatively difficult to support the data expiration mechanism of complicated data structures on a key-value storage engine. Ardb, however, uses special methods to support the expiration mechanism for all data structures.
The specific implementation is as follows:

• The expiration information is stored in the meta value field in the absolute Unix format (ms).
• Based on the above design, TTL/PTTL and other TTL queries only require one meta read.
• Based on the above design, any reads of the metadata will trigger an expiration decision. Since read operations on the metadata are a requisite step, no extra read operations are required here (it is triggered on access like in Redis).
• Create a namespace TTL_DB to specifically store TTL sorting information.
• When the expiration time setting is stored int the metadata and is not zero, an additional key-value will be stored as KEY_TTL_SORT. The key encoding format is [TTL_DB] "" KEY_TTL_SORT, and the value is empty. Therefore, operations similar to expire settings in Ardb will require an additional write operation for implementation.
• In the custom comparator, the key comparison rule for the KEY_TTL_SORT type is to compare first, so that the KEY_TTL_SORT data will be saved in the order of the expiration time.
• A thread is started independently in Ardb to scan the KEY_TTL_SORT data in order at regular intervals (100 ms). When the expiration time is earlier than the current time, the delete operation will be triggered. When the expiration time is later than the current time, the scan will be terminated (equivalent to the timed serverCron task in Redis for processing expired keys).

Concluding remarks

Through the conversion on the encoding layer, we can well encapsulate KVStore for integration with Redis. All operations on Redis data, after the conversion on the encoding layer, will eventually be converted to n reads and writes (N> = 1) to the KVStore. On the premise of conforming to the Redis command semantics, we tried to minimize the number of "n"s in encoding.
The most important point is that the integration of Redis and KVStore is not meant to replace Redis, but to enable the single machine to support a data size that far exceeds the memory capacity while maintaining an acceptable level of performance. In a particular situation, it can also serve as a cold data storage solution to achieve interconnection with the hotspot data in Redis.

Community

Redis+KVStore: Disk-based Storage for Massive Data

Encoding Mode

Key Encoding Format

Value Encoding Format

Encoding of various data types

ZSet encoding example

Implementation of Del

Implementation of Expire

Concluding remarks

Read previous post:

Read next post:

Alibaba Clouder

You may also like

Comments

Raja_KT February 12, 2019 at 4:23 am

Alibaba Clouder

Related Products

ApsaraDB for Redis