Redis's newest version, Redis 5.0 RC1, has already been released. The most important feature of the new version is Redis Streams. Let's quickly look at what Redis Streams is and discuss its applications in real-world scenarios.
Redis Streams is a feature that has been planned for many years. Redis Streams was originally planned for version 4.0, but because it is a relatively heavy feature and the kernel changes are also relatively large, it has been postponed to Redis 5.0.
Redis Streams is essentially a message queue, but it is also unique compared to other message middleware such as Kafka and RocketMQ. It is, by nature, a message publishing and subscription component on Redis kernel (non Redis Module).
Although the existing PUB/SUB and BLOCKED LIST can be used as a message queue service in simple scenarios, Redis Streams is more comprehensive. It provides message persistence and Master/slave data replication functions; it also has the new RadixTree data structure to support more efficient memory use and message reading. Redis Streams also supports a function similar to Kafka's Consumer Group.
Our focus in this article is on how to use Redis Streams in practical business scenarios.
I'm sure most of you are familiar with Internet Relay Chat (IRC) platforms. Many well-known open source projects (including Redis) have their own IRC channels for convenient live idea exchange between developers and users. Interestingly enough, the topic of this article, Redis Streams, was originated from a discussion on IRC!
A typical IRC model is as follows:
A user in an IRC channel can send and receive messages to and from all other users without restrictions. To build a Redis-based IRC system, we will naturally think of Redis's PUB/SUB function.
We can see that when using the PUB/SUB function, we can receive messages sent by all users (clients) if all users have subscribed to the same IRC channel (channel1). To send messages, we can simply use the
publish command. The entire business logic is very clear and straight forward. This is also an important reason why Redis is so powerful and popular—it provides functions and data structures that greatly improve developers' efficiency.
However, the PUB/SUB-based IRC has a problem: PUB/SUB's uses the so-called "Fire-and-Forget" message model. That is to say Redis does not save any message history.
If a user in an IRC channel is disconnected due to network failure and rejoins the IRC channel later on, he will not be able to see the chat log for the period when he is disconnected. New users also cannot see the historical chat log for the latest discussion. This causes great inconvenience for users who want to quickly understand the currently discussed topics. In addition, if Redis is restarted, then all users need to re-subscribe to their channels again.
What if we create IRC channels based on Redis Streams?
Create a Channel
# Currently Redis does not support creating empty streams. We can add a special message, # to create a new stream (channel) ip:7000> xadd channel1 * create-channel null 1528702126345-0
# Use the xadd command to send a message. We can name each message and attach the message source for business logic processing convenience. # We can also send multiple messages at one time as this is helpful for optimizing network overhead. ip:7000> xadd channel1 * msg1-tony "Hello everyone." 1528702503377-0 ip:7000> xadd channel1 * msg2-tony "I am a big Redis fan." msg3-tony "Hope we can learn from each other.:-)" 1528702573546-0
# If a new user joins a channel for the first time, "$" is specified as a special start ID for message reading. The user will then only receive the channel's latest messages. # If there are new messages, the user will start reading the messages from the previous return result ID. # If there are no new messages, the xread command returns an empty set ip:7000> xread BLOCK 100 STREAMS channel1 $ 1) 1) "channel1" 2) 1) 1) 1528703048021-0 2) 1) "msg1-tony" 2) "Hello everyone." ip:7000> xread BLOCK 100 STREAMS channel1 1528703048021-0 1) 1) "channel1" 2) 1) 1) 1528703061087-0 2) 1) "msg2-tony" 2) "I am a big Redis fan." 3) "msg3-tony" 4) "Hope we can learn from each other.:-)" ip:7000> xread BLOCK 100 STREAMS channel1 1528703061087-0 (nil)
Retrieve Historical Messages
We have mentioned that, an important difference between Redis Streams and PUB/SUB is that Redis Streams allows users to receive historical messages. Therefore, if a user rejoins an IRC channel after disconnection, the following methods can be used to allow this user to retrieve historical messages:
# 1528703061087-0 is the ID of the user's last received message ip:7000> xrange channel1 1528703061087-0 + 1) 1) 1528706457462-0 2) 1) "msg1-andy" 2) "Nice to meet you guys." 2) 1) 1528706497200-0 2) 1) "msg4-tony" 2) "When will Redis 5.0 GA comes out?" 3) 1) 1528706601973-0 2) 1) "msg1-antirez" 2) "I think it will arrive in the second half of 2018."
In addition to its powerful and extensive data structure support, Redis also is also important for cross-platform systems because of its cross-platform compatibility. Redis can even be used as a built-in storage system to run on an ARM-based platform. Redis can even run on Raspberry Pi.
In the Internet of Things (IoT) era to come, there will be numerous smart devices ready to be connected to the internet, anytime and anywhere. For example, home appliances will be interconnected to your mobile phone. Your refrigerator could report to the quantity and freshness of food; your air conditioner could report to you the room temperature and air quality; and your car could continuously report to you various parameters about your engine, your gearbox, and the car's internal environment. With such a large number of IoT devices interacting, this will definitely generate a huge data flow. These data will be collected and analyzed on the cloud and could potentially generate huge user value.
These types of data may be different in various aspects; the similarity is that all such data is time series data. Now, you may suddenly recall that Redis Streams was designed to support time series data (refer to Redis Streams introduction in the first section). Moreover, Redis can be successfully run on ARM devices. It is predicted that trillions of ARM-powered IoT devices will appear in the future. Therefore, we can imagine that, apart from its wide application in various Internet services as Cache and KV storage, Redis will soon flourish in the IoT industry.
The above diagram shows IoT devices' typical information collection, analysis, and display architecture. Redis is running on various IoT devices as a built-in storage system. Each device uses Redis Streams to temporarily store the generated time series data and then asynchronously push the same to the cloud. Then, business programs on the cloud will retrieve the raw data, analyze the data based on certain rules, and write the results to reliable data storage systems. At last, users retrieve the results and display the same on their apps or web pages. Altogether, the system forms a closed loop.
Redis Streams is one of the most exciting features of Redis 5.0. One of the biggest uses of Redis Streams is for the Internet of Things industry. There is no doubt that Redis Streams will be an important component for the Fourth Industrial Revolution (4IR).
To learn more about Redis on Alibaba Cloud, visit the ApsaraDB for Redis product page.
MongoShake – A MongoDB-based Cross-Data Center Data Replication Platform
Alibaba Cloud Native Community - March 20, 2023
Alibaba Cloud Security - January 16, 2020
ApsaraDB - July 10, 2019
Apache Flink Community China - September 27, 2020
Alibaba Cloud Native Community - December 26, 2022
William Pan - August 19, 2019
Provides secure and reliable communication between devices and the IoT Platform which allows you to manage a large number of devices on a single IoT Platform.Learn More
Block-level data storage attached to ECS instances to achieve high performance, low latency, and high reliabilityLearn More
A cloud solution for smart technology providers to quickly build stable, cost-efficient, and reliable ubiquitous platformsLearn More
Build a Data Lake with Alibaba Cloud Object Storage Service (OSS) with 99.9999999999% (12 9s) availability, 99.995% SLA, and high scalabilityLearn More
More Posts by ApsaraDB
Raja_KT February 12, 2019 at 4:08 am
Good one but if we can see comparable features with popular Kafka...like in ScalabilityHigh-Volume,Data Transformations,Fault Tolerance,Reliability,Durability,PerformanceZero Downtime , then we can recourse to using it.