From Messaging Service to Cloud Native Event Streaming Platform

History of Apache RocketMQ

Reviewing the development of Apache RocketMQ in the past ten years, it can be divided into two stages: "born on the Internet" and "grown in cloud computing".

The first stage is the implementation of RocketMQ from 0 to 1 in scale within Alibaba. In 2012, in order to support the ultra-large-scale e-commerce Internet architecture, Alibaba middleware developed RocketMQ and opened source at the beginning of the product's birth. In 2017, RocketMQ unified the Alibaba messaging technology system.

The second stage is cloud computing. RocketMQ was launched into the cloud in 2015, which is also the first open source message queue in the industry to provide public cloud SaaS form; In 2016, Alibaba donated RocketMQ to Apache, and graduated in 2017, becoming the first TLP Internet middleware in China.

A sword sharpened in ten years will be sharp when it is unsheathed. During the past decade, RocketMQ has always adhered to the development idea of integrating open source, group and business, and the core evolution and product iteration have been going hand in hand through the group's polishing stability, relying on cloud computing incubation and innovation, and open source co-construction to accelerate the standardization establishment and ecological connection. The official release of RocketMQ 5.0 in 2022 announced that it was fully entering the cloud native era.

RocketMQ 5.0: from message service to cloud native event flow platform

Looking back over the past decade, RocketMQ has served almost all the businesses of the Group. On Alibaba Cloud, RocketMQ has served more than 100000 enterprise customers, covering more than 20 industries such as the Internet, retail, finance and automobile. Large-scale production practices continue to accumulate the core advantages of products.

• Diversity, the complex business demands of enterprise-level applications, give rise to RocketMQ providing rich message types, such as timing messages, transaction messages, sequence messages, etc. In addition, it also provides a series of message governance capabilities such as message trace and message backtracking.

• Consistency. Both Taobao transaction and Ant payment naturally have high requirements for data consistency. The distributed transaction message provided by RocketMQ is the first message product in the industry to provide this feature. It perfectly integrates asynchronous decoupling and data consistency, and is an indispensable product capability among financial customers.

• Stability, which is the foundation of the product, is also a system engineering. RocketMQ has been polishing in the e-commerce and financial fields for a long time. It not only provides up to 99.99% SLA, but also helps customers provide comprehensive health inspection and troubleshooting capabilities, such as message tracking, message backtracking, dead-letter mechanism, and so on, providing diversified means of stability.

• High-performance. Under the limit flow of the "Double 11", RocketMQ has unlimited scalability and supports tens of millions of concurrent and trillions of message peaks. P9999 write latency is within 1ms and 100% is within 100ms.

It can be said that in the field of messaging, especially in the field of business messaging, RocketMQ has become the top in China and the first choice for enterprise customers.

With the arrival of cloud native and digital era, RocketMQ is also changing rapidly. What are the main aspects of the change?

First of all, fully embrace cloud primitives. Next, the messaging system itself realizes the evolution of the cloud native architecture, fully releases the pooling capability of the cloud infrastructure, and comprehensively improves the core technical indicators of messaging. Up, the form of messaging products continues to evolve and become the core engine of cloud native application architecture. Such as microservices, event-driven, serverless and other modern application architectures.

Second, embrace real-time data in an all-round way. The digital transformation of enterprises has shifted from the original business digitalization to digital business. Real-time insight into business data and real-time decision-making become the key elements to guide business success. Message queue will also extend from the infrastructure of online business architecture to the infrastructure of real-time data architecture, so as to realize the integration of transaction analysis.

With the release of 5.0, RocketMQ also officially upgraded from message service to cloud native event flow processing platform.

RocketMQ 5.0: Cloud native architecture upgrade

First, let's look at the evolution of RocketMQ's own cloud native architecture. It can be seen from the following panoramic view that RocketMQ has undergone all-round transformation from the client to the server, which is embodied in the following aspects:

1. Lightweight. RocketMQ 4.0 was born in Taobao e-commerce in 2012. At that time, most of the business was still running on the physical machine. The single-node computing capacity was strong, and the number of client nodes was small and relatively stable. Therefore, the access method of rich clients was not only more efficient, but also could provide a series of enterprise-level features such as client-side load balancing, message caching, and failover. However, this mode has changed in the cloud native era, and lightweight clients are more easily integrated by the cloud native technology stack. Therefore, RocketMQ 5.0 client uses the lightweight SDK design concept to sink the logic of the original rich client to the server side, to meet the trend of modern application lightweight, serverless and mesh, and is easier to be integrated; At the same time, it is because of lightweight that the cost of multilingual development of the SDK is much lower, and the current mainstream multilingual versions are quickly covered.

2. Resilience. The storage and computing separation architecture enables stateless computing nodes to scale rapidly, while the hierarchical storage and the cold and hot separation architecture make the message storage more resilient.

3. High availability, based on the new Leaderless architecture, can achieve flexible selection of replica number, synchronous and asynchronous automatic escalation, and achieve second-level failover while eliminating ZK dependency; Cloud-oriented multi-availability zones and multi-regions form a global high-availability capability.

4. Finally, the overall architecture of RocketMQ moves towards Kubernetes, embracing OpenTelemetry, and relying on the ARMS, Prometheus and Grafana provided by Alibaba Cloud to realize the cloud biochemistry of observable capability.

The upgrade of RocketMQ 5.0 has made significant improvements in product capability and cost optimization, in addition to the technical architecture of cloud biochemistry. Let's break it down one by one.

Light and stateless consumption model

RocketMQ 4.0 adopts the consumption model by queue, and consumers are fully balanced according to the queue load. It is very suitable for the scenario of batch pull and fast consumption, which is not sensitive to the status of a single message, such as flow computing. However, in the field of business messaging, especially under the financial scenario and event-driven architecture, the status of each message is extremely important. In addition, the message processing time of different business types is also different. From millisecond level, to second level and even to minute level, the unbalanced load of the queue or the short block may cause local accumulation of messages, thus affecting the end user experience. Therefore, RocketMQ 5.0 launched a new lightweight stateless consumption model based on message load, and skillfully built a message model on top of the queue model through the PoP mechanism. The business only needs to care about the message and does not need to care about the queue. All APIs can support single-message level control, such as message consumption, retry, deletion, etc. Based on the message consumption model, the client, connection and consumption are stateless, and can float on any proxy node, truly achieving lightweight.

RocketMQ 5.0 provides a consumption model by queue and a consumption model by message to better meet the business scenarios of events and flows.

Massive message hierarchical storage

Another major upgrade of RocketMQ 5.0 is the hierarchical storage of massive messages. Students who know about message queues know that messages are usually stored for a short time. Most message products in the industry give priority to the retention time of messages, ranging from 3 days to 7 days, and up to 15 days. The limited storage space not only limits the retention time of messages, but also may cause business loss in some scenarios. For example, when messages are not consumed, they are cleared due to insufficient disk space or expired messages, which is unacceptable in financial and other fields. Therefore, RocketMQ has always wanted to solve such problems and make storage more flexible.

RocketMQ 5.0 builds a cold-hot separation and hierarchical storage architecture based on ESSD and object storage, provides low-cost unlimited storage capacity, and ensures that messages will not be cleared in advance due to insufficient local disk space, resulting in business losses. We provide serverless for message storage. Customers only need to pay for the actual storage usage, instead of pre-ordering storage space.

In addition, traffic peak clipping is a very common scenario for message queues, and the resulting massive message accumulation capacity and performance stability under accumulation conditions become the core indicators to measure product performance. RocketMQ 5.0 further achieves read and write isolation based on the cold and hot data separation architecture to avoid affecting the write performance of hot data in the scenario of accumulation. The cold data fragmentation of hierarchical storage improves the cold data reading performance and provides users with consistent cold reading SLA.

Sales series are upgraded, with a maximum cost reduction of 50%

From the previous introduction, we have learned that RocketMQ 5.0 has significantly improved its technical architecture and product capabilities.

Compared with the billing strategy, the new sales form launched in 5.0 is simpler, more flexible and more inclusive. The comprehensive cost of an instance can be reduced by up to 50%. The minimum access threshold can be reduced to 390 yuan/month, far lower than the self-built cost. Message storage supports the flexibility of Serverless, and pay-as-you-go can significantly reduce idle costs. Combined with the multi-level storage capacity of cold and hot separation, it can reduce 67% compared with open source self-built, and significantly reduce the use cost of message queues.

EventBridge: cloud event hub

Event-driven is a concept with a very early origin. As early as a few decades ago, event-driven technology was widely used in both the design of the operating system kernel and the client programming framework. The API provided by RocketMQ PushConsumer is actually an event-driven programming paradigm, but in the microservice application architecture, integration and communication are just necessary, and the value of event-driven is not so obvious.

With the advent of the cloud native era, the composition of computing power is becoming more and more diverse. As the representative technology of cloud native, the Serverless architecture paradigm is also event-driven. Whether Alibaba Cloud's function computing or AWS's Lambda, their main trigger sources are all kinds of events. Cloud product events, such as OSS file upload, trigger users to process files based on functions; User business events, such as RocketMQ trigger function running consumption logic processing, etc.

With event-driven as the core concept, Alibaba Cloud launched the EventBridge product, whose mission is to build an event hub on the cloud. Four business values can be realized through EventBridge:

1. Unify the event hub. Alibaba Cloud has hundreds of millions of events every day, from IaaS, PaaS to third-party SaaS, but there is no simple and unified way to reach these events; These events are scattered in various places to form "islands of events", and it is difficult to excavate useful business value. Only by giving full play to the scale effect of data and establishing the blood relationship between data can we better explore the value of data; Therefore, the first task of EventBridge is to unify the event specifications on Alibaba Cloud, embrace the CloudEvents event standards, and build a unified event hub for Alibaba Cloud.

2. Event-driven engine. When the EventBridge is connected to a large number of event sources, based on the RocketMQ millisecond level event triggering capability, it will certainly accelerate the enterprise EDA/Serverless architecture upgrade.

3. Open and integrated. EventBridge provides rich cross-cloud, cross-platform, cross-product, cross-region and cross-account connectivity capabilities, which can promote the integration of cloud products, applications and SaaS services.

4. Low code. With the help of the application center of Serverless, EventBridge can achieve event distribution, filtering, transformation and other processing through simple rule matching and rich templates, further improving the efficiency of event-driven.

Let news be everywhere and events be everywhere

Relying on the strong combination of EventBridge, RocketMQ and functional computing FC, the event ecology of EventBridge has begun to take shape.

In terms of cloud product event integration, 200+cloud product event sources and more than 3000 event types have been integrated.

In terms of data integration and processing, EventBridge is deeply integrated with micro-service applications, big data, IoT and other scenarios. For example, with the integration of message ecology, Alibaba Cloud's six message queue products realize the interconnection of message data through EventBridge, and provide the ability of global message routing through EventBridge's global event network. At the same time, they can also achieve ETL data processing capability through the rich templates provided by EventBridge.

In terms of SaaS application integration, including Nail, Jushita and more than 50 SaaS services on the cloud can be connected to EventBridge through EventBridgehook.

In terms of event triggering, EventBridge has reached more than 10 event targets, and a large number of event sources can trigger more than 10 event target cloud products including FC/SAE through EventBridge. In addition, at present, EventBridge has docked with Alibaba Cloud's full cloud product API, and any event can be reached through the cloud product API.

In the future, more event sources and event targets will be connected to the EventBridge.

RocketMQ Streams: a new choice for lightweight computing

As mentioned at the beginning, based on the comprehensive upgrade of cloud native architecture, RocketMQ 5.0 will also extend from the infrastructure of online business architecture to the infrastructure of real-time data architecture to achieve the integration of transaction analysis. Make RocketMQ Streams a new choice for lightweight computing.

Most of the most common big data frameworks in the industry, such as Flink and Spark, adopt the centralized Master-Slave architecture, which is heavily dependent and deployed. The execution of each task requires a large amount of overhead and has a high cost of use. The difference is that RocketMQ Streams focuses on creating a lightweight computing engine with light resources and high performance, without additional dependency. It can be deployed at the minimum of 1g core. It is suitable for scenarios with large amount of data, high filtering, and light window computing. In resource sensitive scenarios, such as message queue stream computing, security risk control, edge computing, RocketMQ Streams has great advantages. Alibaba Cloud's messaging team works with the security team to improve the performance by 3-5 times and save 50% - 80% of resources through a lot of optimization of filtering scenarios.

At present, RocketMQ Streams has been released in the open source community. In the future, it is planned to complete commercialization in Alibaba Cloud in April 2023.

RocketMQ, we are moving forward together

RocketMQ has made many achievements after more than ten years of polishing. The world has more than 700 contributors, 1.8w Stars, and more than 80% of mainstream cloud vendors have provided RocketMQ's commercial hosting services. The Apache RocketMQ community has always maintained a high level of activity. Therefore, it has also won more than 10 domestic and foreign open source awards, such as the "Open Source Innovation List" of Science and Technology Innovation China, the China, Japan and South Korea Open Source Software Outstanding Technology Award.

Alibaba Cloud, as the origin and core contributor of RocketMQ, not only covers 100% of the whole group's business, but also carries tens of millions of concurrent trillions of messages. Over the past decade, it has served more than 10 million corporate customers, covering more than 20 industries such as the Internet, retail and automobile. More than 75% of the head enterprises choose to use RocketMQ. It is expected that Alibaba Cloud's message queue RocketMQ will become the choice of enterprise customers. We also invite more developers to actively participate in the construction of RocketMQ's open source community and build RocketMQ as a leader in the field of messaging.