Redis Streams – Redis 5.0's Newest Data Type

Redis's newest version, Redis 5.0 RC1, has already been released. The most important feature of the new version is Redis Streams. Let's quickly look at what Redis Streams is and discuss its applications in real-world scenarios.

What Is Redis Streams?

Redis Streams is a feature that has been planned for many years. Redis Streams was originally planned for version 4.0, but because it is a relatively heavy feature and the kernel changes are also relatively large, it has been postponed to Redis 5.0.

Redis Streams is essentially a message queue, but it is also unique compared to other message middleware such as Kafka and RocketMQ. It is, by nature, a message publishing and subscription component on Redis kernel (non Redis Module).

Although the existing PUB/SUB and BLOCKED LIST can be used as a message queue service in simple scenarios, Redis Streams is more comprehensive. It provides message persistence and Master/slave data replication functions; it also has the new RadixTree data structure to support more efficient memory use and message reading. Redis Streams also supports a function similar to Kafka's Consumer Group.

Our focus in this article is on how to use Redis Streams in practical business scenarios.

Redis Streams – IRC Systems

I'm sure most of you are familiar with Internet Relay Chat (IRC) platforms. Many well-known open source projects (including Redis) have their own IRC channels for convenient live idea exchange between developers and users. Interestingly enough, the topic of this article, Redis Streams, was originated from a discussion on IRC!

A typical IRC model is as follows:

A user in an IRC channel can send and receive messages to and from all other users without restrictions. To build a Redis-based IRC system, we will naturally think of Redis's PUB/SUB function.

We can see that when using the PUB/SUB function, we can receive messages sent by all users (clients) if all users have subscribed to the same IRC channel (channel1). To send messages, we can simply use the publish command. The entire business logic is very clear and straight forward. This is also an important reason why Redis is so powerful and popular—it provides functions and data structures that greatly improve developers' efficiency.

However, the PUB/SUB-based IRC has a problem: PUB/SUB's uses the so-called "Fire-and-Forget" message model. That is to say Redis does not save any message history.

If a user in an IRC channel is disconnected due to network failure and rejoins the IRC channel later on, he will not be able to see the chat log for the period when he is disconnected. New users also cannot see the historical chat log for the latest discussion. This causes great inconvenience for users who want to quickly understand the currently discussed topics. In addition, if Redis is restarted, then all users need to re-subscribe to their channels again.

What if we create IRC channels based on Redis Streams?

Create a Channel

# Currently Redis does not support creating empty streams. We can add a special message,
# to create a new stream (channel)
ip:7000> xadd channel1 * create-channel null
1528702126345-0

Send Messages

# Use the xadd command to send a message. We can name each message and attach the message source for business logic processing convenience.
# We can also send multiple messages at one time as this is helpful for optimizing network overhead.
ip:7000> xadd channel1 * msg1-tony "Hello everyone."
1528702503377-0
ip:7000> xadd channel1 * msg2-tony "I am a big Redis fan." msg3-tony "Hope we can learn from each other.:-)"
1528702573546-0

Receive Messages

# If a new user joins a channel for the first time, "$" is specified as a special start ID for message reading. The user will then only receive the channel's latest messages.
# If there are new messages, the user will start reading the messages from the previous return result ID.
# If there are no new messages, the xread command returns an empty set
ip:7000> xread BLOCK 100 STREAMS channel1 $
1) 1) "channel1"
   2) 1) 1) 1528703048021-0
         2) 1) "msg1-tony"
            2) "Hello everyone."
ip:7000> xread BLOCK 100 STREAMS channel1 1528703048021-0
1) 1) "channel1"
   2) 1) 1) 1528703061087-0
         2) 1) "msg2-tony"
            2) "I am a big Redis fan."
            3) "msg3-tony"
            4) "Hope we can learn from each other.:-)"
ip:7000> xread BLOCK 100 STREAMS channel1 1528703061087-0
(nil)

Retrieve Historical Messages

We have mentioned that, an important difference between Redis Streams and PUB/SUB is that Redis Streams allows users to receive historical messages. Therefore, if a user rejoins an IRC channel after disconnection, the following methods can be used to allow this user to retrieve historical messages:

# 1528703061087-0 is the ID of the user's last received message
ip:7000> xrange channel1 1528703061087-0 +
1) 1) 1528706457462-0
   2) 1) "msg1-andy"
      2) "Nice to meet you guys."
2) 1) 1528706497200-0
   2) 1) "msg4-tony"
      2) "When will Redis 5.0 GA comes out?"
3) 1) 1528706601973-0
   2) 1) "msg1-antirez"
      2) "I think it will arrive in the second half of 2018."

Redis Streams – IoT Data Connections

In addition to its powerful and extensive data structure support, Redis also is also important for cross-platform systems because of its cross-platform compatibility. Redis can even be used as a built-in storage system to run on an ARM-based platform. Redis can even run on Raspberry Pi.

In the Internet of Things (IoT) era to come, there will be numerous smart devices ready to be connected to the internet, anytime and anywhere. For example, home appliances will be interconnected to your mobile phone. Your refrigerator could report to the quantity and freshness of food; your air conditioner could report to you the room temperature and air quality; and your car could continuously report to you various parameters about your engine, your gearbox, and the car's internal environment. With such a large number of IoT devices interacting, this will definitely generate a huge data flow. These data will be collected and analyzed on the cloud and could potentially generate huge user value.

These types of data may be different in various aspects; the similarity is that all such data is time series data. Now, you may suddenly recall that Redis Streams was designed to support time series data (refer to Redis Streams introduction in the first section). Moreover, Redis can be successfully run on ARM devices. It is predicted that trillions of ARM-powered IoT devices will appear in the future. Therefore, we can imagine that, apart from its wide application in various Internet services as Cache and KV storage, Redis will soon flourish in the IoT industry.

The above diagram shows IoT devices' typical information collection, analysis, and display architecture. Redis is running on various IoT devices as a built-in storage system. Each device uses Redis Streams to temporarily store the generated time series data and then asynchronously push the same to the cloud. Then, business programs on the cloud will retrieve the raw data, analyze the data based on certain rules, and write the results to reliable data storage systems. At last, users retrieve the results and display the same on their apps or web pages. Altogether, the system forms a closed loop.

Summary

Redis Streams is one of the most exciting features of Redis 5.0. One of the biggest uses of Redis Streams is for the Internet of Things industry. There is no doubt that Redis Streams will be an important component for the Fourth Industrial Revolution (4IR).

To learn more about Redis on Alibaba Cloud, visit the ApsaraDB for Redis product page.

Community

Redis Streams – Redis 5.0's Newest Data Type

What Is Redis Streams?

Redis Streams – IRC Systems

Redis Streams – IoT Data Connections

Summary

Read previous post:

Read next post:

ApsaraDB

You may also like

Comments

Raja_KT February 12, 2019 at 4:08 am

ApsaraDB

Related Products

IoT Platform

IoT Solution

Data Lake Storage Solution

Cloud Data Transfer