Watch the replay of the Apsara Conference 2021 at this link!
Message Queue is the communication infrastructure of contemporary applications and the core dependencies of microservices applications. It enables users to build distributed, high-performance, elastic, and robust applications more efficiently through asynchronous decoupling.
From the perspective of data and business value, the value of Message Queue is deepening gradually. The core business data flowing in Message Queue involves different aspects and scenarios, such as integrated transmission, analytical computing, and processing. With continuous evolution, we can foresee the new value and chemical reactions created by Message Queue in scenarios, such as data channel, event-driven integration, and analytical computing.
RocketMQ originated from Alibaba's internal e-commerce system. Its core architecture has undergone many key stages of evolution from inception to the present day:
In 2007, when Taobao's e-commerce system was split into different services, the first generation message service, Notify, was born, which was the earliest form of RocketMQ. Notify used relational databases for storage with a push mode. It has a wide range of applications in high-frequency trading scenarios, such as Taobao.
From 2007 to 2013, with the development of Alibaba Group's business, asynchronous calls in trading scenarios and the transmission of a large number of tracking point data and data synchronization were required. At that time, MetaQ and RocketMQ 3.0 were derived internally. These two versions began to explore the proprietary storage engine and adopted in-house proprietary message storage. They supported the massive number of topics on a single server and removed the external dependencies of components, such as ZooKeeper, with foresight. Ten years later, we see that removing various keepers has become the mainstream development direction across the whole message realm.
After the previous three generations were tested by internal businesses, Alibaba actively opened and contributed RocketMQ 3.0 to the open-source community. In 2017, RocketMQ 3.0 officially graduated from Apache Incubator and became China's first top-level project in the Apache community outside the Hadoop ecosystem. Since then, RocketMQ has also officially started to serve the enterprise clients of Alibaba Cloud. Following the unified development strategy of open-source commercialization and internalization, version 4.x was released in 2018 and focused on high reliability and low latency optimization. It realized a new low-latency storage engine and multi-scenario disaster recovery solution and provided rich messaging attributes. This also makes RocketMQ the preferred solution for financial-grade business messages.
Last month, the community released RocketMQ 5.0 Preview, officially announcing the arrival of 5.0. RocketMQ 5.0 is no longer limited to the basic scenario of message decoupling. Now, it provides processing capabilities, featuring the integration of messages, events, and streams through the advantages of unified kernel and storage.
After more than a decade of development on RocketMQ, its good community environment and business support enable a large number of enterprise developers to follow up business features and demands for selection and authentication easily. In terms of influence among active communities, the RocketMQ community has gained 15,000+ stars, with 400+ active contributors and 30+ active projects in areas, such as multi-language and eco-connectivity. RocketMQ is popular among community developers. In terms of application scale, RocketMQ is a financial-grade business messaging solution. It has accumulated tens of thousands of enterprise clients from many industries, such as online gaming, online education, finance and security, banking, government and enterprise energy, and automobile transportation. At the same time, it is responsible for Alibaba's core business process. It circulates trillions of messages every day and has handled the midnight peak spikes during every previous Double 11 Shopping Festival. In terms of industry evaluation, RocketMQ has also won many top prizes.
Alibaba Cloud Message Queue for Apache RocketMQ 5.0 was released recently in China. We call it an overall integrated processing platform for messages, events, and streams.
The latest version has two new highlights. The first one is the expansion and layout of the core scenario of messages. RocketMQ 5.0 is no longer limited to the message decoupling scenario but will apply to event-driven and message stream processing scenarios. The second one is the technical architecture and trend of overall integrated processing.
The technical architecture of overall integrated processing for messages, events, and streams enables single-message storage for message scenarios, such as streaming computing, asynchronous delivery, and integration driving. This reduces the technical complexity and O&M costs of business personnel in the O&M of multiple systems significantly. The unified RocketMQ 5.0 product stack can handle microservice instruction calls, asynchronous notifications, CDC change logs, behavior tracking point data, resource O&M, and audit events.
First of all, the most important upgrade is the comprehensive renewal of the technical architecture of Alibaba Cloud RocketMQ.
The new RocketMQ 5.0 expands the unified storage logic, which centrally solves the technical problems of message storage, such as multiple duplicates, low latency, and massive queue partitions. It processes the upper layer messages and strips a completely stateless computing layer from them. It mainly supports protocol adaptation, permission management, consumption states, and observable O&M systems. Thanks to the architecture design that separates storage from computing, the comprehensive procedure from SDK access to online O&M is improved.
In addition to the improvement of capabilities in the message basic architecture, Alibaba Cloud RocketMQ has launched the lightweight message ETL function in the streaming processing scenario.
Users generate messages in scenarios, such as database changes, terminal data reporting, and background tracking point logs. In a typical consumption scenario, data is cleaned, converted, and stored in external storage and offline or online analysis systems. Traditional solutions require building heavyweight real-time processing services, such as Flink, or user-created consumption applications for message processing. With the RocketMQ ETL function, data cleansing and conversion can be realized by simply configuring them in the console. The RocketMQ ETL function has three major advantages:
The last highlight of the new RocketMQ is the layout and evolution in event-driven business scenarios. In 2018, Gartner's report listed Event-Driven-Architecture (EDA) as one of the top ten strategic technology trends. EDA will become the mainstream part of microservices in the future. Let's start with a definition:
Event-driven is essentially an upgrade of message-driven, which is the next inevitable stage in the deep evolution of enterprise IT architecture.
The differences and associations between event-driven architectures and message-driven architectures lie in the following three points:
Therefore, Alibaba Cloud EventBridge, a Unified Event Center, brings the following capabilities:
The release of RocketMQ 5.0 marks Alibaba Cloud messages officially moving from the messaging realm to a new situation featuring the deeply integrated message, event, and stream scenarios. In the future, the development of Alibaba Cloud messaging products will continue to focus on the core scenarios of messages, events, and streams. The basic architecture of messages will also continue to move forward in the direction of serverless elasticity, strong disaster recovery capability, observability, and no O&M, bringing high-SLA services with high performance, high reliability, and strong disaster recovery to customers. In the streaming scenario, more message processing and computing services will be launched continuously by combining with ecological products based on clients' business demands. We will create a future-oriented enterprise integration mode, cooperate with ecological partners and open-source communities, and develop EDA further.
Seven Major Technological Innovations and Breakthroughs of Alibaba Cloud in the Serverless Field
How to Run a Ghost Blog using Alibaba Cloud Simple Application Server
1,014 posts | 247 followers
FollowAlibaba Cloud Native Community - January 5, 2023
Alibaba Cloud Native Community - March 20, 2023
Alibaba Cloud Native Community - December 16, 2022
Alibaba Cloud Native Community - November 23, 2022
Alibaba Cloud Native - June 12, 2024
Alibaba Cloud Native - June 12, 2024
1,014 posts | 247 followers
FollowApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn MoreA message service designed for IoT and mobile Internet (MI).
Learn MoreA distributed, fully managed, and professional messaging service that features high throughput, low latency, and high scalability.
Learn MoreAlibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.
Learn MoreMore Posts by Alibaba Cloud Community