>> Watch the replay of the Apsara Conference 2022 at this link!
Speaker: Qingshan Lin (Longji) (Co-Founder of Apache RocketMQ, Senior Technical Expert of Alibaba Cloud, and Head of Alibaba Cloud's Message Product Line) is an expert in the international messaging field, committed to the research and exploration of messaging, real-time computing, and event-driven architecture, promoting the evolution of RocketMQ cloud-native architecture and hyper-converged infrastructure.
Over the past 20 years, open-source technology prospered with the rapid development of the Internet and cloud computing. The following figure shows that the amount of investment in open-source projects is increasing year by year. In the era of consumer Internet, the explosive growth of Internet business has stimulated the first wave of open-source. Entering the industrial Internet era, the vigorous development of cloud computing has led to the second wave of open-source.
In the digital era, the rise of open-source technology and cloud computing both share the underlying logic of improving the productivity of the software industry by transforming production relations. Open-source technology has changed the software development model. You can quickly obtain feedback from developers through the construction of open-source communities. At the same time, popular open-source projects can attract hundreds (or thousands) of developers to co-develop and build together, which significantly improves the iteration speed of open-source software. Open-source technology has also broken the technology chimney and contributed to the standardization of software, which has further improved the efficiency of industrial collaboration.
Cloud computing has changed the way software is delivered, making infrastructure and basic software service-oriented. Customers can use them out of the box and pay for them on a pay-as-you-go basis. It has reduced costs, improved efficiency for enterprise digitalization, and accelerated the process of the digital economy.
Just as new energy has brought about changes in the automotive industry, cloud computing has brought technological innovation opportunities for basic open-source software. More open-source software adopts cloud-native architecture and releases cloud computing dividends, bringing breakthroughs in technological competitiveness:
Elasticity
Cloud computing provides elastic computing, storage, and network capabilities. Basic software should implement the elastic architecture and seamlessly combine the elastic computing power of the cloud to release elastic dividends.
Cost
The cloud infrastructure layer is deeply optimized based on the combination of software and hardware to achieve optimal costs. Basic software should sink more capabilities to the infrastructure layer, such as making full use of cloud storage to reconstruct storage modules and release low-cost dividends.
High Availability
Cloud computing provides a global multi-region and multi-zone infrastructure. The basic software should have the ability to synchronize data and status across multiple zones to release global high availability.
Next, let's use RocketMQ as an example to talk about how cloud computing accelerates RocketMQ innovation. First, let's review the development history of Apache RocketMQ over the past ten years. It can be divided into two stages: born on the Internet and grown on cloud computing.
The first stage is RocketMQ from 0 to 1, being implemented on a large scale within Alibaba. In 2012, Alibaba middleware developed RocketMQ to support the ultra-large-scale e-commerce Internet architecture and opened its source at the beginning of the product's birth. In 2017, RocketMQ unified the Alibaba messaging technology system.
The second stage is cloud computing. In 2015, RocketMQ was migrated to the cloud. This is also the industry's first open-source message queue to provide public cloud (SaaS). In 2016, Alibaba donated RocketMQ to Apache, and RocketMQ graduated in 2017, becoming Internet middleware as the first TLP in China.
Under the combination of cloud computing and open-source technology, RocketMQ has been implemented on a large scale outside Alibaba, helping thousands of industries complete digital transformation and further leaps in product capabilities. With Version 5.0 being released in 2022, Apache RocketMQ officially entered the cloud-native era.
Apache RocketMQ has created four core advantages for large-scale e-commerce and financial scenarios.
The first advantage is the ability to support ultra-large-scale and complex businesses with rich message features. Every large Internet company has its main business (such as the transaction service of Alibaba, the payment service of Ant Group, and the take-out service of Eleme) and expands its business capabilities centered on its main business. The Alibaba e-commerce platform is an e-commerce operating system built around transaction events. Each transaction event will trigger different businesses, and different segments of businesses will focus on different types of transaction events. For example, the vertical market only focuses on the transaction events of a certain category, the Tmall Mart only focuses on the transaction events of a certain seller, and the shopping cart only focuses on the transaction events of successful orders. The SQL subscription of RocketMQ provides the most flexible message filtering capability, which can meet the demands of downstream consumers to filter messages according to different business dimensions. In addition to their main businesses, large Internet companies will quickly iterate and diversify their innovative businesses. A large number of small-traffic topics will be generated for message queues. RocketMQ has the capability of 10,000-level throughput of topics in the standalone mode and can maximize resource utilization to provide multi-rent services. There are also various timing event-triggered scenarios in large-scale Internet businesses, the most typical of which is the order closing mechanism for transaction timeout. Timed messages of RMQ can easily meet such demands.
The second advantage is consistency. Both Alibaba transactions and Ant Group payments have high requirements for data consistency. RocketMQ also provides multiple key features in terms of consistency. The most representative is the distributed transaction message. RocketMQ is the first message queue to implement this feature. This feature can ensure that upstream and downstream transactions are ultimately consistent with the order status. This scheme has also become the de facto standard of the asynchronous message consistency scheme, which has been adopted by many Internet companies. Some companies have transplanted it to the customized version of Kafka. In addition to the data consistency in the distributed system, sequential consistency is required in some scenarios, such as the heterogeneous database replication scenario generated by sharding. For example, for the data replication of Taobao buyers and sellers, RocketMQ can distribute the binlogs of the source database to the destination database in strict order for data replay, thus achieving data consistency in different business dimensions.
In the case of high concurrent traffic on the Internet, applications make extensive use of caching technology. In addition to centralized caching services, applications adopt local caches to accelerate further. The broadcasting message of RocketMQ can broadcast cache invalidation to all machines of the application, implementing the consistency of distributed local caches.
The third advantage is stability. This is the foundation of transaction and financial scenarios. Stability is a systematic project. RocketMQ has the core HA capability but also provides an all-around stability guarantee (such as the adaptive pull mode), which prevents slow consumers from being crushed and truly realizes load shifting. Message trace (the industry's first message queue with its own message-level observability) allows users to locate and troubleshoot online problems accurately and quickly. Message backtracking and message dead letter mechanisms provide a variety of stability methods for consumption failures caused by business bugs or data problems.
The fourth advantage is high performance. In the Double 11 Shopping Festival scenario, 99.995% of RocketMQ message write latency is within 1 ms and 100% within 100 ms. RocketMQ uses the shared-nothing distributed architecture and has unlimited scalability in terms of throughput. It has supported trillion-level message peaks during the Double 11 Shopping Festival for ten consecutive years, providing low-latency message services for million-level application instances.
RocketMQ is evolving in two directions for cloud-native in the cloud computing era. RocketMQ product forms are evolving upward to better support the cloud-native application architecture (Microservice, Event-Driven Architecture (EDA), and Serverless). RocketMQ is transforming its cloud-native architecture to fully release the elasticity of cloud infrastructure and comprehensively improve key technical indicators.
Next, let's first look at the evolution in the product form of RocketMQ based on the cloud-native application architecture:
Microservice is the core of the cloud-native application architecture. The introduction of the microservices model allows digital enterprises to collaborate efficiently in a high cohesion, low coupling manner based on miniaturized business units and teams. However, the microservices model also brings new problems. For example, a large number of synchronous microservices will cause risks (such as increased latency and reduced availability). The introduction of the message queue to build an asynchronous microservice system can improve the resilience of microservices and reduce latency.
Several de facto standards have been formed in the field of microservices (such as the Spring Cloud system and Dubbo system). At the same time, the next-generation microservice technology is developing rapidly, which is mainly reflected in the sinking of infrastructure, such as service mesh. It decouples the implementation of business logic and distributed infrastructure so that both of them can evolve independently and flexibly. If the infrastructure sinks further, it will evolve in the direction of Serverless, and the service granularity will become smaller to the function level. Code development and O&M are decoupled, and developers do not need to pay attention to the O&M of applications and can spend more time on business development.
Facing the trend of microservices, RocketMQ 5.0 provides lightweight SDKs, and a large number of features are implemented on the server (such as message retries and load balancing). Lightweight SDKs are more friendly to mesh technology. Now, the mesh capability of RocketMQ is integrated into the official CNCF Envoy community. Lightweight SDKs use the message-granularity load balancing mechanism. The client does not need to bind queues for consumption, which implements stateless consumption and matches the scenarios of Serverless applications better.
Today, based on the core capabilities of RocketMQ 5.0, various cloud messaging products (such as Alibaba Cloud RocketMQ, MNS, and RabbitMQ) are supported to help customers build modern application architectures.
The application scope of event-driven architecture has expanded in the cloud-native era, becoming one of Gartner's top 10 technology trends of the year. 60% of the new digital business solutions adopt EDA.
Event-driven architecture is a classic concept. As early as a few decades ago, both the design of the operating system kernel and the client programming framework adopted a large number of event-driven technologies. The value of EDA is the ability to provide decoupling for software design. Decoupling is the core for the implementation of rapid software iteration. The value of EDA will be fully released in the industry-wide digital transformation and full cloud migration era.
The enterprise business in the real world can be abstracted into a process of event triggering and event response one by one. For example:
A: If a commodity is purchased in a supermarket, an event that user A purchases a certain commodity will be generated. Responses from the supermarket to this event may trigger the commodity replenishment action.
B: After completing the deposit in ATM, the event that A deposits XX yuan will be generated, and the bank will respond to this event, which may trigger the recommendation actions of financial products.
Use events to think about business and conduct business modeling and then use event-driven technology to conduct business digitalization construction, which can build highly scalable and highly agile digital systems for enterprises. For example, in the case of the supermarket above, the event of purchasing commodities can trigger more actions in the future, such as the construction of the CRM system, placement optimization of commodity shelves, coupon issuance, etc. The digital system based on EDA will continue to create business increments.
On the other hand, business opportunities in the real world are often fleeting. Event-driven business systems are naturally triggered in real-time and respond quickly to businesses. For example, on e-commerce websites, through real-time analysis of users' click events and shopping cart events on the website, the business system analyses user preferences and needs in real-time and provides an accurate real-time recommendation, improving the shopping success rate and transaction volume.
After the large-scale digital transformation of the entire industry, cross-business and cross-organization business cooperation will move from offline to online. The scale of digital business ecology will continue to expand in the digital economy era. In the past, cross-organization business collaboration was mostly carried out through synchronous API calls. Now, with the expansion of scale, the increase of participants, and the acceleration of business iteration, the synchronous API call method is far from meeting the requirements of digital business ecosystem development. There are significant problems, including cross-organization businesses that cannot evolve and iterate independently due to coupling, availability degradation, and the performance failing to meet the scale of 2C.
The natural asynchronous and decoupling features of EDA can solve this series of problems, and cross-organization business collaboration needs to be completely decoupled. For example, Alibaba Cloud Tmall adopts the event-driven architecture. Tmall releases transaction events in real-time, and partners (including ISV, software service providers, and brand merchants), subscribe and consume transaction events and build the personalized CRM, merchant operation, and background management system, forming a huge e-commerce digital ecosystem. DingTalk starts the strategy of Cloud + DingTalk integration, opens the comprehensive event source of DingTalk through DingTalk connector + EventBridge, and builds a digital office ecosystem with industry partners in an event-integrated manner. The event-driven architecture will break business silos and data silos for a new cross-organization business ecosystem. The industry-level event specification CloudEvent has also been formed, and the connection will create new value in the future.
As a cloud-native representative technology, the Serverless architecture is also event-driven. Currently, the main product forms of Serverless architecture (Alibaba Cloud Function Compute (FC)) are triggered by various forms of events. For cloud product events, the upload of an OSS file triggers the process and calculation of files based on functions. For user business events, EventBroker triggers functions to run consumption logic. For cloud product O&M events, users can expand their automated O&M system on the cloud platform according to the event response.
The large-scale use of the event-driven architecture can help digital enterprises unleash the technical dividends of Serverless. After the large-scale cross-organization implementation of EDA, CNCF introduced the CloudEvent specification. Based on the unified specification, there is a common language for cross-system and cross-organization digital collaboration, which enables more efficient system integration and precipitates event-oriented software infrastructure.
In response to the trend of EDA, RocketMQ 5.0 has released a new product form called Eventbridge. Its domain model is event-centric, and it supports CloudEvent specifications and provides low-code event orchestration, filtering, routing capabilities, and flexible event integration capabilities.
Today, based on the core capabilities of RocketMQ 5.0, it supports Alibaba Cloud EventBridge to help cloud customers implement an event-driven and event-integrated business ecosystem.
Now, let's look at the evolution of cloud-native architecture from the perspective of RocketMQ technology architecture. This is a panoramic view of the cloud-native architecture of RocketMQ. The all-around transformation has been carried out from the client to the server with higher elasticity, availability, and lower cost.
Proxy is a completely stateless computing node. It focuses on multi-protocol and multi-domain scenarios and can be elastic for different workloads. For example, different scenarios of IoT, microservices, and big data have different resource demands. At the same time, the topic resources of RocketMQ are decoupled in three layers: message-oriented topics, stream-oriented logical shards of topics, and underlying storage-oriented physical shards of topics. Each layer can be independently scaled.
The Store layer focuses on highly available storage of messages, including replica replication, primary/secondary switchover, and cloud storage integration. The new HA technology is introduced into the Leaderless architecture, the identity of Store nodes is peer, and the resource utilization rate is improved in an all-around way. It maintains a minimalist architecture with built-in distributed consistency components and does not need to rely on distributed systems (such as external Zookeeper). The multi-replica strategy can be customized. Users can flexibly combine availability + reliability + cost in different scenarios. Based on the new highly available architecture, users can use cloud-oriented multi-zone and multi-region to form Geo global high availability.
In terms of cloud storage integration, RocketMQ 5.0, in the form of cloud services, can build a multi-level storage architecture based on cloud disks and Object Storage Service (OSS), provide unlimited storage capabilities at low costs, separate hot and cold data, and provide consistent cold reading SLA for users.
After seven years of large-scale cloud computing practice, RocketMQ has entered the 5.0 era. From the original message middleware for the Internet business to a message, event, and stream hyper-converged processing platform, it unlocks more comprehensive capabilities.
As shown in the following figure, IoT devices continuously generate data, and RocketMQ at the edge directly performs real-time data analysis and calculation, which can quickly respond to users. At the same time, high-value events or data generated by real-time ETL and real-time decision-making can also be transmitted to the cloud. They can be linked to the cloud platform through the Eventing capability of RocketMQ, making full use of the one-stop platform technology of the public cloud to amplify the value of data. Now, RocketMQ can connect everything and run anywhere, providing a cloud, edge, and terminal integrated real-time data solution.
Next, we will continue increasing investment in the ecological construction of RocketMQ 5.0. One is the construction of the application architecture ecosystem, which includes both classic open-source projects and standard integrations (such as JMS and AMQP) as well as the integrated cloud-native technology ecosystems (such as CloudEvents, Dapr, and Envoy).
At the same time, RocketMQ will continue developing the data architecture ecosystem to integrate big data ingestion, data storage, data processing, and data analysis components from offline big data to real-time big data. RocketMQ delivers a complete business value chain to customers by linking with community ecological projects to achieve business coverage in multiple scenarios (such as event-driven architecture solutions and real-time data solutions).
With the joint construction of 700+ contributors worldwide, the product competitiveness of RocketMQ keeps improving, and RocketMQ has been recognized by more users. Recently, RocketMQ has won many awards, including the excellent software of China Open Source Cloud League, the Innovation China Open Source Innovation List, and the Apache China Open Source Project Leaders Quadrant.
As the number of potential users increases, the commercial value of RocketMQ is further enlarged. Currently, more than ten cloud vendors provide commercial services of Apache RocketMQ with no vendor locked-in. RocketMQ has become the common choice of the Internet, finance, and cloud vendors and is the de facto standard in the messaging field.
In the future, RocketMQ will continue evolving around the cloud-native message queue that integrates messages, events, and streams. This is a key feature of recent RocketMQ planning. Community developers are welcome to participate in joint construction.
An Analysis of the Basic Concept of "Message-Driven, Event-Driven, and Streaming"
How to Protect Gateway High Availability in Big Promotion Scenarios
481 posts | 48 followers
FollowAlibaba Cloud Community - November 25, 2022
Alibaba Cloud Community - December 9, 2021
Alibaba Cloud Native Community - December 6, 2022
Alibaba Clouder - October 10, 2018
Clouders - October 17, 2022
Alibaba Cloud Community - December 21, 2021
481 posts | 48 followers
Follow
ApsaraMQ for RocketMQ
ApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn More
Function Compute
Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.
Learn More
Elastic High Performance Computing Solution
High Performance Computing (HPC) and AI technology helps scientific research institutions to perform viral gene sequencing, conduct new drug research and development, and shorten the research and development cycle.
Learn More
Quick Starts
Deploy custom Alibaba Cloud solutions for business-critical scenarios with Quick Start templates.
Learn MoreMore Posts by Alibaba Cloud Native Community
