Alibaba Cloud Message Queue (MQ) is a distributed message queue service independently developed by Alibaba and fully hosted on the Alibaba Cloud platform. It supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications. This service can be used to easily create a scalable distributed system with loose coupling and high availability.
Message Queue has been integrated into the infrastructure of Alibaba Group for over 11 years and servicing Alibaba’s 11-11 e-commerce sales event for nine years. It features low latency, high concurrency, high availability, high reliability, and powerful message stack capability and supports trillion-level data floods. You can use MQ to freely transmit data between application components at any scale.
In September 2017, RocketMQ, the kernel engine of Alibaba Cloud Message Queue, became an Apache top-level project. Adhering to the principle of openness, sharing, and joint construction, it embraces an open-source ecosystem and has no technology binding. RocketMQ is a next-generation message engine architecture based on OpenMessaging. It is applied primarily to four fields and supports hybrid cloud multi-lingual architecture. It has been adopted by the Linux Foundation and is expected to become an international standard in the distributed message field.
- High-performance and Low Latency
Message Queue supports 10-million-level Queries Per Second (QPS) and trillion-level message flow with no concurrency and upper-performance limits
- High Reliability
Message Queue supports synchronous double-write, ultra-triple copy data redundancy across servers, and rapid switching data copy technology, ensuring data reliability up to 99.99999999%
- High Scalability
Message Queue supports queues of ten thousand messages. The queue number can be elastically scaled and the cluster scale can be automatically scaled. This process is fully transparent to users.
- Permission Control
Message Queue controls the permissions of sending and receiving each message by the message topic and consumer ID, ensuring message security
High-performance and Low Latency
Message Queue ensures high-performance even in the case of a massive message stack and without affecting cluster service. This is particularly important during load shifting (flood storage).
From years of experience servicing Alibaba’s annual 11-11 e-commerce event, the technical team at Alibaba Cloud have developed methods to resolve slow request response time jitter problems caused by system overload due to super-high concurrency. Among the flow of the trillion messages sent during the 11-11 event held in 2017, the recorded latency of 99.6% of message write requests was less than 1ms, and less than 10ms for the recorded latency of 99.996% message write requests.
The message is sent immediately after it arrives at the server, ensuring timely message delivery.
High Availability, Reliability, and Scalability
Message Queue provides highly available message cloud services in multiple regions worldwide. The server rooms and hardware facilities are built according to high standards in accordance with the IDC of Alibaba. Each region is deployed with multiple server rooms. Message Queue can still effectively release messages to applications even if an entire server is out of service.
Message Queue supports same-city multi-active deployment solutions in multiple financial regions including China East 1 (Hangzhou), China East 2 (Shanghai), China South 1 (Shenzhen) and other regions, protecting core financial transaction systems.
Secure Access Control
Message Queue supports primary account and sub-account creation, blacklist and whitelist functionality, and Alibaba Cloud RAM’s STS functions.
Message Queue supports cross-account authorization and authorization through primary accounts and sub-accounts.
The ordered messages are consumed in the order of release (first-in-first-out). Message Queue supports global order and segmented order of messages.
Message Queue supports TLS and access from Alibaba Cloud VPC.
Topic Model (Topic)
Message Queue supports Pub/Sub mode. In this mode, the creator sends the message to the specified topic. One or multiple consumers can subscribe to the message.
Message Queue supports cluster consumption and broadcast consumption. Subscribers that use the same consumer ID belong to the same cluster. In cluster consumption mode, each message only needs to be processed by any one of the subscribers in the cluster. In broadcast consumption mode, each message is pushed to all subscribers in the cluster, ensuring that the message is consumed by each subscriber in the cluster at least once.
Diversified Types of Messages
Common message is used for asynchronous business decoupling between systems, load shifting, logging, large-scale cache synchronization between hosts, and real-time computing and analysis.
Message Queue provides a distributed transaction function known as X/Open XA to ensure transaction consistency.
After a message is sent to the Message Queue server, the Message Queue server sends the message at the specified time point (for example, at 15:00:00 on January 1, 2016) after the message sending time (the current time).
After a message is sent to the Message Queue server, the Message Queue server sends the message at the specified delay time point (for example, 30 minutes later) after the message sending time (the current time).
Systematic Maintenance Tools
Message Queue provides a comprehensive set of maintenance tools to help you quickly detect and process system problems.
Query messages by topic, message ID or message key on the Message Queue server.
Message Queue records complete data about the entire process including the message flow from the creator to the Message Queue server and then to the end-user, aggregates and analyzes the data, and constructs a visual full-link message track.
Message Queue can play back consumed messages at the specified time and thereby acts as a powerful troubleshooting tool.
Message Queue allows you to view historical and real-time data of each message by the topic and consumer ID, helping you to analyze the collected data.
You can set monitoring and alert rules for each customer ID based on the consumption delay time and stack quantity of messages, so as to detect timely problems.
Message Queue provides a complete set of standard HTTP RESTful Open APIs of control type, which can implement a series of resource management and maintenance functions, as well as facilitate access.
- Load Shifting (Flood Storage)
- Distributed Transactions
- Real-time Computing
- Large Scale Cache Synchronization
Loose coupling between systems helps to improve scalability and reliability, and is the best design solution for modern applications. By using Message Queue, you can send, store, and receive messages between any quantity of application service components (systems) with zero message losses. This simplifies the running of cloud application service components (systems) and delivers higher cost-effectiveness.
Load Shifting (Flood Storage)
During massive-scale events, such as the peak period of transactions at midnight during Alibaba’s Tmall 11-11 e-commerce festival, the Spring Festival Gala online red packet money give away, and other major online events, a large number of requests can fail when external request volume exceeds the processing capability of the system. If the system has no appropriate protection measures, when too many accumulated requests time out and cannot be restored automatically, the system collapses and its service capability is deemed as 0 for external services. This, in turn, can adversely and substantially impact the user experience.
In conventional transaction processing mode, business processes between multiple systems or application components are coupled with a large transaction, resulting in a long response time and long business link, thus affecting system availability. After introducing the distributed transaction messages of Message Queue, you can split core link businesses that need to be processed synchronously from branch link businesses that can be processed asynchronously, to split a large transaction into a small transaction to reduce interaction between systems.
In this way, you can ensure data consistency between distributed systems and decouple businesses systems (shopping cart, points, and others) from each other, to realize an optimal architecture design.
Data generates value during "flow". Conventional data analysis is mostly based on the batch computing model, which cannot realize real-time data analysis. By combining Alibaba Cloud Message Queue with a mainstream computing engine (such as Spark, Storm, EMR, ARMS, and BeamRunner), you can conveniently perform real-time business data analysis.
Large Scale Cache Synchronization
For parallel sessions of large activities or a large-scale APP client, large-scale cache design is applied to reduce the request response time (RT) and improve user customer experience. In the case of Tmall's 11-11 promotion, a wide variety of products are presented to customers on the Shoutao app, Tmall app, and various specialty malls, and the price of every product fluctuates in real time as the promotion progresses. A large number of concurrent accesses to the product database page result in long response time, and the centralized cache bandwidth becomes a bottleneck, leading to failure to meet product price access demands. The multi-cache design for specialty malls uses Message Queue to realize synchronization between multiple caches, to allow customers to access the product price.
Within Alibaba, both TradeManager (an app for instant trade communication) and DingTalk (an app for office communication) use Message Queue for their messaging services. A large number of messages are generated during each interaction occurred in 1-on-1 chats, group chats and DingTalk notifications.