Apsara Conference 2022 | Alibaba Cloud Message Queue RocketMQ 5.0
Date: Oct 1, 2022
Introduction: Apsara Conference 2022 the release of RocketMQ 5.0 marks that Alibaba Cloud Messaging has officially moved from the messaging field to a new situation where the "message, event, and flow" scenarios are integrated.
Introduction: Apsara Conference 2022 the Great Convergence From "Messages" to "Messages, Events, Streams"
Apsara Conference 2022 as the communication infrastructure of contemporary applications, message queues are the core dependencies of microservice architecture applications. Through asynchronous decoupling capabilities, users can more efficiently build distributed, high-performance, elastic and robust applications Apsara Conference 2022.
Apsara Conference 2022 from the perspective of data value and business value, the value of message queues continues to deepen. The business core data flowing in the message queue involves different links and scenarios such as integrated transmission, analysis and calculation, and processing. With continuous evolution, we can foresee that message queues will inevitably generate new value and create new "chemical reactions" in scenarios such as data channels, event integration drivers, and analytical computing Apsara Conference 2022.
RocketMQ was born in Alibaba's internal e-commerce system. Today, its core architecture has undergone many key evolutions Apsara Conference 2022:
As early as 2007, when the Taobao e-commerce system was split into services, the first-generation messaging service Notify was born, which is the earliest prototype of RocketMQ. Notify uses a relational database as storage and uses push mode. It is widely used in high-frequency trading scenarios such as Ali Taobao Apsara Conference 2022.
Apsara Conference 2022 during the period from 2007 to 2013, with the development of Alibaba Group's business, not only asynchronous calls of transaction scenarios are required, but also a large amount of buried point data transmission and data synchronization are required. At this time, the MetaQ and RocketMQ3.0 versions were derived internally. Apsara Conference 2022 these two versions began to explore the self-developed storage engine, adopted the self-developed proprietary message storage, supported a large number of topics on a single machine, and proactively removed the external components such as Zookeeper. rely. Today, ten years later, we see that various keepers have become mainstream in the entire messaging field.
After the first three generations of internal business polishing, Alibaba actively participated in open source and contributed RocketMQ3.0 to the open source community, and officially graduated from the Apache incubator in 2017, becoming China's first non-Hadoop ecosystem top-level project in the Apache community. Since then, RocketMQ has also officially started to serve Alibaba Cloud enterprise customers. Adhering to the three-in-one development strategy of open source, business, and internal development, the 4.x version released in 2018 focuses on optimization in terms of high reliability and low latency, builds a new low-latency storage engine and multi-scenario disaster recovery solutions, and provides rich news characteristic. This also makes RocketMQ the preferred solution for financial-grade business messages.
Last month, the community released the RocketMQ 5.0-preview version, officially announcing the arrival of 5.0. RocketMQ5.0 will no longer be limited to the basic scenario of message decoupling, but will provide integrated processing capabilities of messages, events, and streams through the advantages of unified kernel and storage.
Looking back on the more than ten years of RocketMQ development, the good community environment and commercial support have enabled a large number of enterprise developers to easily follow up business characteristics and demands for model selection and verification. In terms of active community influence, the RocketMQ community project has gained 15,000+ Stars, 400+ active contributors, and 30+ surrounding active projects such as multi-language and ecological connections, which are well received by community developers. In terms of application scale, RocketMQ, as a financial-level business messaging solution, has accumulated tens of thousands of corporate customers in many industries such as Internet games, online education, financial securities, banking, government and enterprise energy, and automobile travel. At the same time, it is responsible for the core business links within Alibaba, and it circulates trillion-level message traffic every day, which has overcome the zero-point peak of previous Double Eleven. In terms of industry evaluation, RocketMQ has also won many awards.
Major Release 1: New Upgrade of RocketMQ Infrastructure
First of all, the most important upgrade is the comprehensive renovation of the technical architecture of Alibaba Cloud RocketMQ.
The new RocketMQ 5.0 version sinks the general storage logic, centrally solves technical problems such as multiple copies of message storage, low latency, and massive queue partitioning, and separates the upper-layer message processing and stripping out of a complete stateless computing layer, mainly to complete the protocol. Adaptation, authority management, consumption status, and observable operation and maintenance system support. Thanks to the architecture design of the separation of storage and computing, the whole link from SDK access to online operation and maintenance has brought a comprehensive improvement:
1. The openness of the lightweight version of the SDK and the improvement of the whole-link observable system: it supports both the 4.x communication protocol and the new gRPC communication protocol, and has built-in support for OpenTelemetry buried points. The new version of the SDK has added more than 10 indicators. point.
2. Message-level load balancing: The new version of the SDK no longer participates in the load balancing of the actual storage queue, and the message load balancing will be more lightweight, with a single message as the smallest unit of scheduling.
3. Multi-network access support: The new version supports a single instance to expose public network, intranet and other access forms at the same time, which is convenient for customers to access multiple networks.
4. Massive hierarchical storage: The new version opens the ability to save historical messages in hierarchical storage. The message is low-cost and has no size limit, and can be stored for a maximum of 30 days. The hot and cold data are separated and designed to greatly reduce the performance impact of consuming historical messages on the instance.
Major Release 2: RocketMQ Streaming Best Practices on the Cloud - Message ETL
In addition to improving the capabilities of the messaging infrastructure, Alibaba Cloud RocketMQ has launched a lightweight message ETL function in the Streaming streaming scenario.
A typical consumption scenario for messages generated by users in scenarios such as database changes, terminal data reporting, and back-end logging is data cleaning and transformation, and at the same time, they are stored in external storage and offline analysis and online analysis systems. The traditional implementation scheme needs to build heavyweight real-time computing services such as Flink or self-built consumer applications for message processing. Using the commercial version of RocketMQ ETL function, simple console configuration can achieve message cleaning and conversion. The RocketMQ ETL function has three advantages:
1. Lightweight and no dependencies: As the native function of Alibaba Cloud Messaging, it does not need to deploy external computing services or consumer programs when using it, and the solution is lighter.
2. Low development threshold: Built-in common cleaning and conversion templates meet most of the message content processing needs, and support users to quickly write custom functions to support special business logic. The overall development cost is very low, and the business can be launched in 1 hour.
3. Serverless elasticity: There is no need to estimate the capacity in advance, and the serverless serverless model is adopted to realize elastic scaling on demand.
Major Release 3: Best Practices on EDA Cloud - Event Hub EventBridge
The last release point of RocketMQ this time is the layout and evolution of event-driven business scenarios. As early as 2018, the Gartner assessment report listed EDA (Event-Driven-Architecture) as one of the top ten strategic technology trends, and event-driven architecture will become the mainstream of microservices in the future. We first define the following:
The essence of event-driven is a re-upgrade of message-driven, and it is the next inevitable stage of the in-depth evolution of enterprise IT architecture.
The difference and association between event-driven architecture and message-driven architecture mainly focus on the following three points:
1. EDA places more emphasis on deep-level decoupling: message-driven is the decoupling of calls at the technical architecture level between different components in the same business and organizational system, and its information encapsulation and processing are expected and predefined. Event-driven adaptation is a broader business and organizational system. Event-based decoupling does not require expectations and behavior definitions between upstream and downstream. The upstream and downstream uniformly follow standardized specifications, which is a deeper decoupling.
2. EDA places more emphasis on connection capabilities: message-driven is more of a call within a single system, while event-driven often involves different regions, account entities, and the coordination of three-party SaaS. A major feature of event-driven is the strong ecological connection capability. .
3. EDA puts more emphasis on serverless low-code development: analogous to the synergistic relationship between messages and microservices, the general trend of serverless business architecture in the future will push the business development model to gradually shift to low-code configuration. Another major feature of event-driven is low-code development. Based on rich tool capabilities, the business side does not need to write a lot of production and consumption code like message-driven.
Therefore, the Alibaba Cloud Unified Event Center EventBridge product brings the following capabilities:
1. Unified and standardized event integration ecosystem: As the Alibaba Cloud event center, it integrates business events from more than 80 cloud products, and supports more than 800 event types. Users can use EventBridge to manage changes to all cloud product resources, operate and use events at one time. Avoid the repetitive work of connecting multiple product interfaces.
2. Global Event Interconnection Network: Implementing the attribute capability of event-driven strong connection, the Global Event Interconnection Network was released this time, and the first batch of events supported the five major domestic regional events. Enterprise customers can achieve cross-account, cross-region, and cross-network event aggregation and transfer with simple configuration.
3. Serverless low-code development: There are more than ten built-in event targets and processing templates, covering most business scenarios, with simple configuration and low-code for customers, and event driving and processing can be completed without deploying services.
Facing the future: firmly promoting the development of the integration of "messages, events and streams"
The release of RocketMQ 5.0 marks that Alibaba Cloud Messaging has officially moved from the messaging field to a new situation where "messages, events, and streams" scenarios are integrated. In the future, the evolution of Alibaba Cloud's messaging products will continue to focus on the core scenarios of messages, events, and streams. The messaging infrastructure itself will also continue to move forward in the direction of serverless flexibility, strong disaster tolerance, and observable and free operation and maintenance, bringing customers high SLA services with high performance, high reliability, and strong disaster tolerance; and in Streaming Based on customer business demands, the joint ecological products will continue to launch more message processing computing services; create a future-oriented enterprise integration model, and cooperate with ecological partners and open source communities to vigorously promote event-driven further development.
Introduction: Apsara Conference 2022 the release of RocketMQ 5.0 marks that Alibaba Cloud Messaging has officially moved from the messaging field to a new situation where the "message, event, and flow" scenarios are integrated.
Introduction: Apsara Conference 2022 the Great Convergence From "Messages" to "Messages, Events, Streams"
Apsara Conference 2022 as the communication infrastructure of contemporary applications, message queues are the core dependencies of microservice architecture applications. Through asynchronous decoupling capabilities, users can more efficiently build distributed, high-performance, elastic and robust applications Apsara Conference 2022.
Apsara Conference 2022 from the perspective of data value and business value, the value of message queues continues to deepen. The business core data flowing in the message queue involves different links and scenarios such as integrated transmission, analysis and calculation, and processing. With continuous evolution, we can foresee that message queues will inevitably generate new value and create new "chemical reactions" in scenarios such as data channels, event integration drivers, and analytical computing Apsara Conference 2022.
RocketMQ was born in Alibaba's internal e-commerce system. Today, its core architecture has undergone many key evolutions Apsara Conference 2022:
As early as 2007, when the Taobao e-commerce system was split into services, the first-generation messaging service Notify was born, which is the earliest prototype of RocketMQ. Notify uses a relational database as storage and uses push mode. It is widely used in high-frequency trading scenarios such as Ali Taobao Apsara Conference 2022.
Apsara Conference 2022 during the period from 2007 to 2013, with the development of Alibaba Group's business, not only asynchronous calls of transaction scenarios are required, but also a large amount of buried point data transmission and data synchronization are required. At this time, the MetaQ and RocketMQ3.0 versions were derived internally. Apsara Conference 2022 these two versions began to explore the self-developed storage engine, adopted the self-developed proprietary message storage, supported a large number of topics on a single machine, and proactively removed the external components such as Zookeeper. rely. Today, ten years later, we see that various keepers have become mainstream in the entire messaging field.
After the first three generations of internal business polishing, Alibaba actively participated in open source and contributed RocketMQ3.0 to the open source community, and officially graduated from the Apache incubator in 2017, becoming China's first non-Hadoop ecosystem top-level project in the Apache community. Since then, RocketMQ has also officially started to serve Alibaba Cloud enterprise customers. Adhering to the three-in-one development strategy of open source, business, and internal development, the 4.x version released in 2018 focuses on optimization in terms of high reliability and low latency, builds a new low-latency storage engine and multi-scenario disaster recovery solutions, and provides rich news characteristic. This also makes RocketMQ the preferred solution for financial-grade business messages.
Last month, the community released the RocketMQ 5.0-preview version, officially announcing the arrival of 5.0. RocketMQ5.0 will no longer be limited to the basic scenario of message decoupling, but will provide integrated processing capabilities of messages, events, and streams through the advantages of unified kernel and storage.
Looking back on the more than ten years of RocketMQ development, the good community environment and commercial support have enabled a large number of enterprise developers to easily follow up business characteristics and demands for model selection and verification. In terms of active community influence, the RocketMQ community project has gained 15,000+ Stars, 400+ active contributors, and 30+ surrounding active projects such as multi-language and ecological connections, which are well received by community developers. In terms of application scale, RocketMQ, as a financial-level business messaging solution, has accumulated tens of thousands of corporate customers in many industries such as Internet games, online education, financial securities, banking, government and enterprise energy, and automobile travel. At the same time, it is responsible for the core business links within Alibaba, and it circulates trillion-level message traffic every day, which has overcome the zero-point peak of previous Double Eleven. In terms of industry evaluation, RocketMQ has also won many awards.
Major Release 1: New Upgrade of RocketMQ Infrastructure
First of all, the most important upgrade is the comprehensive renovation of the technical architecture of Alibaba Cloud RocketMQ.
The new RocketMQ 5.0 version sinks the general storage logic, centrally solves technical problems such as multiple copies of message storage, low latency, and massive queue partitioning, and separates the upper-layer message processing and stripping out of a complete stateless computing layer, mainly to complete the protocol. Adaptation, authority management, consumption status, and observable operation and maintenance system support. Thanks to the architecture design of the separation of storage and computing, the whole link from SDK access to online operation and maintenance has brought a comprehensive improvement:
1. The openness of the lightweight version of the SDK and the improvement of the whole-link observable system: it supports both the 4.x communication protocol and the new gRPC communication protocol, and has built-in support for OpenTelemetry buried points. The new version of the SDK has added more than 10 indicators. point.
2. Message-level load balancing: The new version of the SDK no longer participates in the load balancing of the actual storage queue, and the message load balancing will be more lightweight, with a single message as the smallest unit of scheduling.
3. Multi-network access support: The new version supports a single instance to expose public network, intranet and other access forms at the same time, which is convenient for customers to access multiple networks.
4. Massive hierarchical storage: The new version opens the ability to save historical messages in hierarchical storage. The message is low-cost and has no size limit, and can be stored for a maximum of 30 days. The hot and cold data are separated and designed to greatly reduce the performance impact of consuming historical messages on the instance.
Major Release 2: RocketMQ Streaming Best Practices on the Cloud - Message ETL
In addition to improving the capabilities of the messaging infrastructure, Alibaba Cloud RocketMQ has launched a lightweight message ETL function in the Streaming streaming scenario.
A typical consumption scenario for messages generated by users in scenarios such as database changes, terminal data reporting, and back-end logging is data cleaning and transformation, and at the same time, they are stored in external storage and offline analysis and online analysis systems. The traditional implementation scheme needs to build heavyweight real-time computing services such as Flink or self-built consumer applications for message processing. Using the commercial version of RocketMQ ETL function, simple console configuration can achieve message cleaning and conversion. The RocketMQ ETL function has three advantages:
1. Lightweight and no dependencies: As the native function of Alibaba Cloud Messaging, it does not need to deploy external computing services or consumer programs when using it, and the solution is lighter.
2. Low development threshold: Built-in common cleaning and conversion templates meet most of the message content processing needs, and support users to quickly write custom functions to support special business logic. The overall development cost is very low, and the business can be launched in 1 hour.
3. Serverless elasticity: There is no need to estimate the capacity in advance, and the serverless serverless model is adopted to realize elastic scaling on demand.
Major Release 3: Best Practices on EDA Cloud - Event Hub EventBridge
The last release point of RocketMQ this time is the layout and evolution of event-driven business scenarios. As early as 2018, the Gartner assessment report listed EDA (Event-Driven-Architecture) as one of the top ten strategic technology trends, and event-driven architecture will become the mainstream of microservices in the future. We first define the following:
The essence of event-driven is a re-upgrade of message-driven, and it is the next inevitable stage of the in-depth evolution of enterprise IT architecture.
The difference and association between event-driven architecture and message-driven architecture mainly focus on the following three points:
1. EDA places more emphasis on deep-level decoupling: message-driven is the decoupling of calls at the technical architecture level between different components in the same business and organizational system, and its information encapsulation and processing are expected and predefined. Event-driven adaptation is a broader business and organizational system. Event-based decoupling does not require expectations and behavior definitions between upstream and downstream. The upstream and downstream uniformly follow standardized specifications, which is a deeper decoupling.
2. EDA places more emphasis on connection capabilities: message-driven is more of a call within a single system, while event-driven often involves different regions, account entities, and the coordination of three-party SaaS. A major feature of event-driven is the strong ecological connection capability. .
3. EDA puts more emphasis on serverless low-code development: analogous to the synergistic relationship between messages and microservices, the general trend of serverless business architecture in the future will push the business development model to gradually shift to low-code configuration. Another major feature of event-driven is low-code development. Based on rich tool capabilities, the business side does not need to write a lot of production and consumption code like message-driven.
Therefore, the Alibaba Cloud Unified Event Center EventBridge product brings the following capabilities:
1. Unified and standardized event integration ecosystem: As the Alibaba Cloud event center, it integrates business events from more than 80 cloud products, and supports more than 800 event types. Users can use EventBridge to manage changes to all cloud product resources, operate and use events at one time. Avoid the repetitive work of connecting multiple product interfaces.
2. Global Event Interconnection Network: Implementing the attribute capability of event-driven strong connection, the Global Event Interconnection Network was released this time, and the first batch of events supported the five major domestic regional events. Enterprise customers can achieve cross-account, cross-region, and cross-network event aggregation and transfer with simple configuration.
3. Serverless low-code development: There are more than ten built-in event targets and processing templates, covering most business scenarios, with simple configuration and low-code for customers, and event driving and processing can be completed without deploying services.
Facing the future: firmly promoting the development of the integration of "messages, events and streams"
The release of RocketMQ 5.0 marks that Alibaba Cloud Messaging has officially moved from the messaging field to a new situation where "messages, events, and streams" scenarios are integrated. In the future, the evolution of Alibaba Cloud's messaging products will continue to focus on the core scenarios of messages, events, and streams. The messaging infrastructure itself will also continue to move forward in the direction of serverless flexibility, strong disaster tolerance, and observable and free operation and maintenance, bringing customers high SLA services with high performance, high reliability, and strong disaster tolerance; and in Streaming Based on customer business demands, the joint ecological products will continue to launch more message processing computing services; create a future-oriented enterprise integration model, and cooperate with ecological partners and open source communities to vigorously promote event-driven further development.
