Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog AlibabaMQ for Apache RocketMQ 5.0: An Integrated Processing Platform for Messages, Events, and Streams

AlibabaMQ for Apache RocketMQ 5.0: An Integrated Processing Platform for Messages, Events, and Streams

This article recaps the last decade of development on RocketMQ and discusses the release of RocketMQ 5.0.

Watch the replay of the Apsara Conference 2021 at this link!

By Alibaba Cloud Developer

From "Message" to the Deep Integration of "Message, Event, and Stream"

Message Queue is the communication infrastructure of contemporary applications and the core dependencies of microservices applications. It enables users to build distributed, high-performance, elastic, and robust applications more efficiently through asynchronous decoupling.

From the perspective of data and business value, the value of Message Queue is deepening gradually. The core business data flowing in Message Queue involves different aspects and scenarios, such as integrated transmission, analytical computing, and processing. With continuous evolution, we can foresee the new value and chemical reactions created by Message Queue in scenarios, such as data channel, event-driven integration, and analytical computing.

1

RocketMQ originated from Alibaba's internal e-commerce system. Its core architecture has undergone many key stages of evolution from inception to the present day:

In 2007, when Taobao's e-commerce system was split into different services, the first generation message service, Notify, was born, which was the earliest form of RocketMQ. Notify used relational databases for storage with a push mode. It has a wide range of applications in high-frequency trading scenarios, such as Taobao.

From 2007 to 2013, with the development of Alibaba Group's business, asynchronous calls in trading scenarios and the transmission of a large number of tracking point data and data synchronization were required. At that time, MetaQ and RocketMQ 3.0 were derived internally. These two versions began to explore the proprietary storage engine and adopted in-house proprietary message storage. They supported the massive number of topics on a single server and removed the external dependencies of components, such as ZooKeeper, with foresight. Ten years later, we see that removing various keepers has become the mainstream development direction across the whole message realm.

After the previous three generations were tested by internal businesses, Alibaba actively opened and contributed RocketMQ 3.0 to the open-source community. In 2017, RocketMQ 3.0 officially graduated from Apache Incubator and became China's first top-level project in the Apache community outside the Hadoop ecosystem. Since then, RocketMQ has also officially started to serve the enterprise clients of Alibaba Cloud. Following the unified development strategy of open-source commercialization and internalization, version 4.x was released in 2018 and focused on high reliability and low latency optimization. It realized a new low-latency storage engine and multi-scenario disaster recovery solution and provided rich messaging attributes. This also makes RocketMQ the preferred solution for financial-grade business messages.

Last month, the community released RocketMQ 5.0 Preview, officially announcing the arrival of 5.0. RocketMQ 5.0 is no longer limited to the basic scenario of message decoupling. Now, it provides processing capabilities, featuring the integration of messages, events, and streams through the advantages of unified kernel and storage.

2

After more than a decade of development on RocketMQ, its good community environment and business support enable a large number of enterprise developers to follow up business features and demands for selection and authentication easily. In terms of influence among active communities, the RocketMQ community has gained 15,000+ stars, with 400+ active contributors and 30+ active projects in areas, such as multi-language and eco-connectivity. RocketMQ is popular among community developers. In terms of application scale, RocketMQ is a financial-grade business messaging solution. It has accumulated tens of thousands of enterprise clients from many industries, such as online gaming, online education, finance and security, banking, government and enterprise energy, and automobile transportation. At the same time, it is responsible for Alibaba's core business process. It circulates trillions of messages every day and has handled the midnight peak spikes during every previous Double 11 Shopping Festival. In terms of industry evaluation, RocketMQ has also won many top prizes.

3

Alibaba Cloud's New Generation of RocketMQ: An Integrated Processing Platform for "Messages, Events, and Streams"

Alibaba Cloud Message Queue for Apache RocketMQ 5.0 was released recently in China. We call it an overall integrated processing platform for messages, events, and streams.

The latest version has two new highlights. The first one is the expansion and layout of the core scenario of messages. RocketMQ 5.0 is no longer limited to the message decoupling scenario but will apply to event-driven and message stream processing scenarios. The second one is the technical architecture and trend of overall integrated processing.

4

The technical architecture of overall integrated processing for messages, events, and streams enables single-message storage for message scenarios, such as streaming computing, asynchronous delivery, and integration driving. This reduces the technical complexity and O&M costs of business personnel in the O&M of multiple systems significantly. The unified RocketMQ 5.0 product stack can handle microservice instruction calls, asynchronous notifications, CDC change logs, behavior tracking point data, resource O&M, and audit events.

Major Release One: New Upgrade of RocketMQ Basic Architecture

First of all, the most important upgrade is the comprehensive renewal of the technical architecture of Alibaba Cloud RocketMQ.

5

The new RocketMQ 5.0 expands the unified storage logic, which centrally solves the technical problems of message storage, such as multiple duplicates, low latency, and massive queue partitions. It processes the upper layer messages and strips a completely stateless computing layer from them. It mainly supports protocol adaptation, permission management, consumption states, and observable O&M systems. Thanks to the architecture design that separates storage from computing, the comprehensive procedure from SDK access to online O&M is improved.

  1. The Openness of the Lightweight SDK and the Improvement of Observable Systems in the Comprehensive Procedure: It supports both the 4.x and new gRPC communication protocols. It also has built-in OpenTelemetry tracking points with more than ten indicator tracking points added to the SDK in the new version.
  2. Message-Level Load Balancing: The new version of the SDK no longer participates in the load balancing of storage queues. Message load balancing will be more lightweight, with a single message as the minimum unit for scheduling.
  3. Support for Multi-Network Access: The new version supports access for a single instance from the Internet and internal networks at the same time, facilitating multi-network access for clients.
  4. Hierarchical Storage for Massive Messages: The new version provides hierarchical storage to save historical messages. The amount of messages is not limited, and messages can be saved for up to 30 days at a low cost. Hot and cold data are designed to be separated to reduce the impact of historical message consumption on the performance of instances.

Major Release Two: Cloud Best Practice of RocketMQ Streaming -- Message ETL

In addition to the improvement of capabilities in the message basic architecture, Alibaba Cloud RocketMQ has launched the lightweight message ETL function in the streaming processing scenario.

6

Users generate messages in scenarios, such as database changes, terminal data reporting, and background tracking point logs. In a typical consumption scenario, data is cleaned, converted, and stored in external storage and offline or online analysis systems. Traditional solutions require building heavyweight real-time processing services, such as Flink, or user-created consumption applications for message processing. With the RocketMQ ETL function, data cleansing and conversion can be realized by simply configuring them in the console. The RocketMQ ETL function has three major advantages:

  1. Lightweight and Dependency-Free: As a native function of Alibaba Cloud messages, there is no need to deploy external computing services or consumption programs to use it, which is more lightweight.
  2. Low Development Threshold: Common built-in templates of cleansing and conversion can meet most of the requirements of message processing. Users can write customized functions to support special business logic quickly. The overall development cost is very low, and the business can be published in one hour.
  3. Serverless Elasticity: No capacity estimation is required. It adopts serverless mode to realize on-demand auto scaling.

Major Release Three: Cloud Best Practice of EDA -- EventBridge

The last highlight of the new RocketMQ is the layout and evolution in event-driven business scenarios. In 2018, Gartner's report listed Event-Driven-Architecture (EDA) as one of the top ten strategic technology trends. EDA will become the mainstream part of microservices in the future. Let's start with a definition:

Event-driven is essentially an upgrade of message-driven, which is the next inevitable stage in the deep evolution of enterprise IT architecture.

The differences and associations between event-driven architectures and message-driven architectures lie in the following three points:

  1. EDA Places More Emphasis on Deep Decoupling: Message-driven is the decoupling of calls between different components in the same business and organizational system at the technical architecture level. Its information encapsulation and processing are expected and predefined. Event-driven adaptation is a broader business and organizational system. Event-based decoupling eliminates the need for expectations and behavior definitions between upstream and downstream. Upstream and downstream follow standardized specifications uniformly, which makes a deeper decoupling.
  2. EDA Places More Emphasis on Connectivity: Message-driven is more of calls within a single system, while event-driven often involves collaboration between different regions, account entities, and third-party SaaS services. A major feature of event-driven is the strong connectivity of the ecosystem.
  3. EDA Places More Emphasis on Serverless Low-Code Development: Compared with the collaborative relationship between messages and microservices, the future trend of serverless business architecture will gradually drive the business development to the low-code configuration mode. Another major feature of event-driven is low-code development. Based on rich tool capabilities, the business side does not need to write a large amount of production and consumption code like message-driven.

7

Therefore, Alibaba Cloud EventBridge, a Unified Event Center, brings the following capabilities:

  1. Unified and Standardized Event Integration Ecosystem: As Alibaba Cloud's event center, it integrates the business events of more than 80 cloud products and supports more than 800 event types. Users can use EventBridge to manage events of changes and operations of all cloud product resources at one time, avoiding repetitive the labor of connecting multiple product APIs.
  2. Interconnected Network of Global Events: The interconnected network of global events was released to implement the attribute capability of strong event-driven connections, which was the first to support event interconnection in five major regions in China. Enterprise clients can realize event aggregation and circulation across accounts, regions, and networks easily with simple configurations.
  3. Serverless Low-Code Development: 10+ built-in event destinations and processing templates cover most of the business scenarios. Clients can simply configure and process events with low code without deploying services.

Face the Future: Steady Promotion of the Deep Integration of "Message, Event, and Stream"

The release of RocketMQ 5.0 marks Alibaba Cloud messages officially moving from the messaging realm to a new situation featuring the deeply integrated message, event, and stream scenarios. In the future, the development of Alibaba Cloud messaging products will continue to focus on the core scenarios of messages, events, and streams. The basic architecture of messages will also continue to move forward in the direction of serverless elasticity, strong disaster recovery capability, observability, and no O&M, bringing high-SLA services with high performance, high reliability, and strong disaster recovery to customers. In the streaming scenario, more message processing and computing services will be launched continuously by combining with ecological products based on clients' business demands. We will create a future-oriented enterprise integration mode, cooperate with ecological partners and open-source communities, and develop EDA further.

Cloud Computing Cloud technology Message Queue Streaming Computing Apsara Conference ETL EventBridge Messaging Eventing Apache RocketMQ RocketMQ 5.0
0 0 0
Share on

Read previous post:

Seven Major Technological Innovations and Breakthroughs of Alibaba Cloud in the Serverless Field

Read next post:

How to Run a Ghost Blog using Alibaba Cloud Simple Application Server

Alibaba Cloud Community

875 posts | 198 followers

You may also like

Comments

Alibaba Cloud Community

875 posts | 198 followers

Related Products

More Posts by Alibaba Cloud Community

See All