Alibaba Cloud Message Queue (MQ) is a distributed message queue service independently developed by Alibaba and fully hosted on the Alibaba Cloud platform. It supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications. This service can be used to easily create a scalable distributed system with loose coupling and high availability.
High-performance and Low Latency
● Message Queue supports 10-million-level Queries Per Second (QPS) and trillion-level message flow with no concurrency and upper-performance limits.
● Message Queue ensures high-performance even in the case of a massive message stack and without affecting cluster service. This is particularly important during load shifting (flood storage).
● From years of experience servicing Alibaba’s annual 11-11 e-commerce event, the technical team at Alibaba Cloud have developed methods to resolve slow request response time jitter problems caused by system overload due to super-high concurrency. Among the flow of the trillion messages sent during the 11-11 event held in 2017, the recorded latency of 99.6% of message write requests was less than 1ms, and less than 10ms for the recorded latency of 99.996% message write requests.
● The message is sent immediately after it arrives at the server, ensuring timely message delivery.
High Availability, Reliability, and Scalability
● Message Queue provides highly available message cloud services in multiple regions worldwide. The server rooms and hardware facilities are built according to high standards in accordance with the IDC of Alibaba. Each region is deployed with multiple server rooms. Message Queue can still effectively release messages to applications even if an entire server is out of service.
● Message Queue supports synchronous double-write, ultra-triple copy data redundancy across servers, and rapid switching data copy technology, ensuring data reliability up to 99.99999999%
● Message Queue supports queues of ten thousand messages. The queue number can be elastically scaled and the cluster scale can be automatically scaled. This process is fully transparent to users.
● Message Queue supports same-city multi-active deployment solutions in multiple financial regions including China East 1 (Hangzhou), China East 2 (Shanghai), China South 1 (Shenzhen) and other regions, protecting core financial transaction systems.
Secure Access Control
● Message Queue controls the permissions of sending and receiving each message by the message topic and consumer ID, ensuring message security.
● Message Queue supports primary account and sub-account creation, blacklist and whitelist functionality, and Alibaba Cloud RAM’s STS functions.
● Message Queue supports cross-account authorization and authorization through primary accounts and sub-accounts.
● The ordered messages are consumed in the order of release (first-in-first-out). Message Queue supports global order and segmented order of messages.
● Message Queue supports TLS and access from Alibaba Cloud VPC.
Message Queue has been integrated into the infrastructure of Alibaba Group for over 11 years and servicing Alibaba’s 11-11 e-commerce sales event for nine years. It features low latency, high concurrency, high availability, high reliability, and powerful message stack capability and supports trillion-level data floods. You can use MQ to freely transmit data between application components at any scale.
In September 2017, RocketMQ, the kernel engine of Alibaba Cloud Message Queue, became an Apache top-level project. Adhering to the principle of openness, sharing, and joint construction, it embraces an open-source ecosystem and has no technology binding. RocketMQ is a next-generation message engine architecture based on OpenMessaging. It is applied primarily to four fields and supports hybrid cloud multi-lingual architecture. It has been adopted by the Linux Foundation and is expected to become an international standard in the distributed message field.
Topic Model (Topic)
● Message Queue supports Pub/Sub mode. In this mode, the creator sends the message to the specified topic. One or multiple consumers can subscribe to the message.
● Message Queue supports cluster consumption and broadcast consumption. Subscribers that use the same consumer ID belong to the same cluster. In cluster consumption mode, each message only needs to be processed by any one of the subscribers in the cluster. In broadcast consumption mode, each message is pushed to all subscribers in the cluster, ensuring that the message is consumed by each subscriber in the cluster at least once.
Diversified Types of Messages
● Common message is used for asynchronous business decoupling between systems, load shifting, logging, large-scale cache synchronization between hosts, and real-time computing and analysis.
● Message Queue provides a distributed transaction function known as X/Open XA to ensure transaction consistency.
● After a message is sent to the Message Queue server, the Message Queue server sends the message at the specified time point (for example, at 15:00:00 on January 1, 2016) after the message sending time (the current time).
● After a message is sent to the Message Queue server, the Message Queue server sends the message at the specified delay time point (for example, 30 minutes later) after the message sending time (the current time).
Systematic Maintenance Tools
● Message Queue provides a comprehensive set of maintenance tools to help you quickly detect and process system problems.
● Query messages by topic, message ID or message key on the Message Queue server.
● Message Queue records complete data about the entire process including the message flow from the creator to the Message Queue server and then to the end-user, aggregates and analyzes the data, and constructs a visual full-link message track.
● Message Queue can play back consumed messages at the specified time and thereby acts as a powerful troubleshooting tool.
● Message Queue allows you to view historical and real-time data of each message by the topic and consumer ID, helping you to analyze the collected data.
● You can set monitoring and alert rules for each customer ID based on the consumption delay time and stack quantity of messages, so as to detect timely problems.
● Message Queue provides a complete set of standard HTTP RESTful Open APIs of control type, which can implement a series of resource management and maintenance functions, as well as facilitate access.
Pre-payment (Subscription): This billing model is mainly applicable to the Message Queue Enterprise Platinum edition. For complete billing details, please see Message Queue Platinum Edition.
Post-payment (Pay-As-You-Go): The details for this billing model are described as follows:
Pay-As-You-Go Billing Items
MQ Fee = API Call Fee + Topic Resource Usage Fee
Number of API Calls = Number of Calls of APIs for Message Sending + Number of Calls of APIs for Message Subscription + Number of Calls of APIs for Long Polling. Long polling refers to the API call generated by the Message Queue end-user for real-time message pushing. Each queue is long polled once every 15 seconds. The long polling count is not increased for any message generated in the queue during this period of time.
Pay-As-You-Go Billing Guide
API Call Fee(Unit Of Measure:USD/1 Million Calls)
|Tiered Billing||Number of Calls/Monthly Calls||HongKong|
|Tier 1||0 to 1 billion||0.45||0.42|
|Tier 2||1 billion to 5 billion||0.41||0.38|
|Tier 3||5 billion to 10 billion||0.34||0.31|
|Tier 4||10 billion to 50 billion||0.3||0.27|
|Tier 5||Over 50 billion||0.27||0.25|
- Topic Resource Usage Fee(Unit Of Measure:USD/Topic/day)
|Tiered Billing||Number of Calls Per Day Per Topic||HongKong|
|Tier 1||0 to 1 million||0.45||0.42|
|Tier 2||1 million to 5 million||0.34||0.31|
|Tier 3||5 million to 10 million||0.11||0.11|
|Tier 4||Over 10 million||0||0|
1. The cost of a post-paid MQ instance is settled on an hourly basis, and the bill is issued every 24 hours (issued on the next day). The fee is automatically deducted from the account balance.
2. The topic resource usage fee is billed in a tiered manner based on the API call count by each topic per day. In master-master account authorization mode, the two master accounts are both billed; while in master-slave account authorization mode, the total sum on the bill is charged to the master account. Therefore, delete the topics as soon as they are no longer in use to avoid unnecessary billing.
3. All the requests to call, send, and receive APIs are billed, and the billing unit is per one million requests. The number of API calls includes the number of calls of APIs for long polling. When messages are normally sent and received, no extra call of long-polling APIs is produced.
4. The maximum message body size is 4 MB. Every 4 KB of published or subscribed data is billed as one request. For example, when the size of one load (published or subscribed) is 256 KB, the API call is billed as 256/4 requests.
5. One request to publish a transaction is billed as 50 requests and one request to subscribe to a transaction is billed as 1 request. For example, when a transaction is published once and subscribed once, it is billed as 50+1=51 API requests.
6. One request to either publish or subscribe an ordered message is billed as 25 requests. For example, when an ordered message is published once and subscribed once, it is billed as 25+25=50 API requests.
7. One request to publish a scheduled message is billed as 25 requests, and one request to subscribe to it is billed as 1 request. For example, when a scheduled message is published once and subscribed once, it is billed as 25+1=26 API requests.
8. The billing coefficient of a financial cloud region is 1.9 times that of a public cloud region.
Free Call Description
1. The first 20 million API calls every month accumulated from from all regions are free of charge.
1. The billing cycle of the Message Queue instance is 24 hours. Thus, on the next natural day, Alibaba Cloud calculates the usage fee of your MQ instance for the previous day, issues a bill, and deducts the usage fee from your Alibaba Cloud account balance accordingly. A bill is generally issued within 8 to 10 hours after the current billing cycle is completed.
2. When your account balance is insufficient to pay off the bill amount and the Message Queue instance has been outstanding for more than 72 hours, Alibaba Cloud ceases providing service and releases the MQ instance. In addition, you cannot access the Message Queue console or APIs. Unprocessed message data will be automatically released, and the released data is unrecoverable.
3. When your pre-paid instance (a Message Queue enterprise platinum edition or an LMQ) expires, Alibaba Cloud stops providing service to you. If the instance fails to be renewed within 168 hours, it will be automatically released, and the released instance and data cannot be recovered.
Message Queue is widely applied across industries including e-commerce, finance, big data, and the Internet of Things (IoT).
Loose coupling between systems helps to improve scalability and reliability, and is the best design solution for modern applications.
By using Message Queue, you can send, store, and receive messages between any quantity of application service components (systems) with zero message losses. This simplifies the running of cloud application service components (systems) and delivers higher cost-effectiveness.
Load Shifting (Flood Storage)
During massive-scale events, such as the peak period of transactions at midnight during Alibaba’s Tmall 11-11 e-commerce festival, the Spring Festival Gala online red packet money give away, and other major online events, a large number of requests can fail when external request volume exceeds the processing capability of the system. If the system has no appropriate protection measures, when too many accumulated requests time out and cannot be restored automatically, the system collapses and its service capability is deemed as 0 for external services. This, in turn, can adversely and substantially impact the user experience.
In conventional transaction processing mode, business processes between multiple systems or application components are coupled with a large transaction, resulting in a long response time and long business link, thus affecting system availability. After introducing the distributed transaction messages of Message Queue, you can split core link businesses that need to be processed synchronously from branch link businesses that can be processed asynchronously, to split a large transaction into a small transaction to reduce interaction between systems.
In this way, you can ensure data consistency between distributed systems and decouple businesses systems (shopping cart, points, and others) from each other, to realize an optimal architecture design.
Data generates value during "flow". Conventional data analysis is mostly based on the batch computing model, which cannot realize real-time data analysis. By combining Alibaba Cloud Message Queue with a mainstream computing engine (such as Spark, Storm, EMR, ARMS, and BeamRunner), you can conveniently perform real-time business data analysis.
Large Scale Cache Synchronization
For parallel sessions of large activities or a large-scale APP client, large-scale cache design is applied to reduce the request response time (RT) and improve user customer experience. In the case of Tmall's 11-11 promotion, a wide variety of products are presented to customers on the Shoutao app, Tmall app, and various specialty malls, and the price of every product fluctuates in real time as the promotion progresses. A large number of concurrent accesses to the product database page result in long response time, and the centralized cache bandwidth becomes a bottleneck, leading to failure to meet product price access demands. The multi-cache design for specialty malls uses Message Queue to realize synchronization between multiple caches, to allow customers to access the product price.
Within Alibaba, both TradeManager (an app for instant trade communication) and DingTalk (an app for office communication) use Message Queue for their messaging services. A large number of messages are generated during each interaction occurred in 1-on-1 chats, group chats and DingTalk notifications.
This section briefly describes how to get started with Message Queue, helping you to better use and understand the product's functions.
Message Queue Quick Access Flowchart
1. Activate the Service
Public cloud users can follow these steps to activate Message Queue:
Navigate to the Alibaba Cloud homepage (https://www.alibabacloud.com/), move the mouse to Product > Internet Middleware, and click MQ to go to the Message Queue product page.
Click Activate Now to go to the Message Queue activation page. Activate product as prompted.
If you have already activated the service, go to the MQ Console .
2. Create Resources
In the Message Queue message system, the message publisher sends a message to a specified topic, and the message subscriber can obtain and consume the message by subscribing to the specified topic.
For more information, please see Create Resources.
3. Send a Message
After applying for a topic resource on the console, you can send messages from the console or call the SDK/API to send messages. Console-based message sending is mainly for fast verification of the availability of the topic resource. In the production environment, you are recommended to call relevant SDK/API to send messages using Message Queue.
For more information, please see Send Messages.
4. Subscribe to Messages
After a message is successfully sent, you must enable the subscriber to subscribe to the message. This document uses TCP Java SDK as an example to describe how to subscribe to a message by calling relevant protocol and SDK/API in the development language.
For more information, please see Subscribe to Messages.
Message Queue SDK Release Note
For more information about downloading and for Release Notes of Message Queue SDKs in multiple languages please see Message Queue SDK Release Note.
See the following resource regarding detailed information about using Message Queue.
1. Is Message Queue accessible from the public network?
Yes, it is. Message Queue provides exclusive public network clusters for testing. When creating a Topic in the Message Queue console, please select the “Public Network” region, and then the Topic is accessible from the public network, but with low availability.
In the production phase, make sure that you select a production region with high availability and deploy it on Alibaba Cloud ECS to send and receive messages.
2. Where does a new Consumer ID start to consume?
If this Consumer ID is started for the first time, then it ignores messages sent before, namely historical messages, and consumes the messages sent after the Consumer ID is started.
If this Consumer ID is started for the second time, then it consumes from the last consumption.
To consume from a specific time point, use the Reset Consumer Offset function of the Message Queue console to specify a starting point of consumption. Each reset only affects the specified Topic under the specified Consumer ID, rather than other Consumer IDs.
3. How to retry after a failed attempt to consume a message?
Clustering consumption: If the consumption business logic code returns Action.ReconsumerLater or NULL, or throws an exception, then it retries up to 16 times. If it still fails after 16 attempts, then the message is discarded. Intervals between each retry are as follows:
|Retried Times||Retry Interval|
A message’s retried times can be retrieved by calling message.getReconsumeTimes().
Broadcasting consumption: For broadcasting consumption, a message is consumed at least once, without retrying after a failed attempt to consume.
4. What if a sent message is not received?
Message Queue supports multiple Message Query methods:
* Query by time range and Topic on all messages received by a Topic within a specified period of time.
* Precise query by Topic and Message ID.
* Relatively precise query by Topic and Message Key on messages with the same Message Key.
With these methods, you can find out the content and consumption status of the messages. To trace the time and location of each node in the entire consumption chain of a message, from being sent by the Producer to being received by the Consumer, you can use Message Queue’s Message Tracing function. For more information, see Message Tracing.
5. Is Message Queue always free of duplicate messages?
In most cases, duplicate messages never happen. However, as a distributed message-oriented middleware, Message Queue doesn’t guarantee this in case of network jitters or application processing timeouts, but it guarantees that messages are never lost.