Ordered messages are frequently used in scenarios that require messages to be consumed in the order in which the messages are produced. For example, ordered messages are frequently used in the financial security and e-commerce industries. Compared with ordered messages 1.0, ordered messages 2.0 provide the following benefits: better parallel processing capacity, higher availability, and more balanced message distribution among partitions. This topic describes the concepts, working mechanism, benefits, and usage notes of ordered messages 2.0 in Message Queue for Apache RocketMQ.
What are ordered messages?
In Message Queue for Apache RocketMQ, ordered messages are messages that are sent and consumed in a strict order. Messages in a topic are published and consumed in the first in, first out (FIFO) order. The earliest published message is consumed as the first message.
Ordered messages are classified into partitionally ordered messages and globally ordered messages. For more information about partitionally ordered messages, see Partitionally ordered message. For more information about globally ordered messages, see Globally ordered messages.
Partitionally ordered message
All messages in a specific topic are partitioned based on the Sharding Key. The messages in each partition are published and consumed in the FIFO order. Only the messages in the same partition are published and consumed in a strict order. Messages across partitions do not need to be consumed in order.
- Sharding Key: the key field used in ordered messages to identify partitions in a topic. This field is different from the key of a normal message. Message Queue for Apache RocketMQ routes the messages that contain the same Sharding Key to a partition. The earliest published message in the partition is consumed as the first message.
- Partition: topic-level partitions. Each topic contains one or more partitions. Messages in the same topic are distributed to these partitions. Logical partitions mentioned in this topic are topic-level partitions. For more information, see Terms.
- Physical partition: the physical unit used to store messages. Each physical partition is created on a specified node of a server.
Partitionally ordered messages are used in scenarios in which high performance is required. Messages in each partition must be published and consumed in the FIFO order.
- If a verification code is required for user registration and the user ID is used as the Sharding Key, the messages sent by the same user are published and consumed in the FIFO order.
- In e-commerce scenarios, the order ID is used as the Sharding Key. Messages for the creation, payment, and refund of the same order are published and consumed in the FIFO order.
All internal e-commerce systems in Alibaba Group use partitionally ordered messages to ensure the order and high performance of your business.
Globally ordered messages
All messages in a topic are published and consumed in the FIFO order.
Globally ordered messages are used in scenarios in which the performance requirements are not high and all messages must be published and consumed in the FIFO order.
When securities are processed, the topic is RMB to USD exchange transactions. If the bid prices of multiple transactions are the same, the earliest transaction is processed as the first transaction. In this scenario, the system must publish and consume messages in the topic in the global FIFO order.
How do ordered messages work?Globally ordered messages work in the same way as partitionally ordered messages. The following section describes how to make sure that messages are sent and consumed in the FIFO order in Message Queue for Apache RocketMQ. Partitionally ordered messages are used in the example.
- Message sending
In the preceding figure, messages for Orders A and B are produced in the sequence of A1, B1, A2, A3, B2, and B3. The messages for the same order must be sent and consumed in the same sequence as the sequence in which these messages are produced. For example, the messages for Order A are sent and consumed in the sequence of A1, A2, and A3. Normal messages for Order A may be sent to different queues in polling mode. The original order of the normal messages cannot be retained in the queues. Ordered messages that use the same Sharding Key such as the same order ID are sequentially routed to the same partition in Message Queue for Apache RocketMQ.
- Message storage
In ordered messages 2.0, each logical partition can correspond to multiple physical partitions. When messages are sent to a logical partition in order, the messages are evenly distributed to the physical partitions for load balancing. The messages stored in a physical partition do not need to be in the original order. Message Queue for Apache RocketMQ records the message order in the logical partition and the location of each message in the physical partition.
- Message consumption
Even if the messages in the same logical partition are stored in the original order in different physical partitions, ordered messages 2.0 use the Message Queue for Apache RocketMQ broker to deliver messages to consumers in the original message order of the logical queue. The consumers use a single thread to consume messages of the same Sharding Key. This ensures that the order in which messages are consumed is the same as the order in which the messages are stored. This way, messages are published and consumed in the same order.
- Balanced distribution of messages
One logical partition can correspond to multiple physical partitions. This feature can be used to resolve the following issue: Messages in logical partitions are not evenly distributed due to the performance bottleneck issue in physical partitions. You can use this feature to resolve the issue when globally ordered messages are used.
- Parallel processing and unlimited scaling
The physical partitions of a logical partition can be scaled out without limits if all messages are sorted in order in the logical partition. This greatly improves the parallel message processing performance and ensures business continuity without incurring additional reconstruction costs.
- Recovery from failures within seconds
If a physical partition fails, the system performs a failover to forward messages to other physical partitions in a few seconds. This ensures that your business remains uninterrupted.
Before you use ordered messages 2.0, take note of the following items:
- Ordered messages 2.0 are available only for Message Queue for Apache RocketMQ Enterprise Platinum Edition instances.
- Make sure that the version of the TCP client SDK for Java is ons-client v184.108.40.206.Final or later.
- Each group ID corresponds to a topic type. A group ID cannot be used to send and receive ordered messages and unordered messages.
- We recommend that you prevent queues that contain globally ordered messages from being congested. You can run multiple instances at the same time to ensure business continuity. If the active instance fails, the system performs a failover on another instance to handle the business. This ensures that your business remains uninterrupted. Only one instance is active at a time.
The Message Queue for Apache RocketMQ broker determines the order in which messages are generated based on the order in which the sender uses a single producer to concurrently send messages in a single thread. If the sender uses multiple producers or multiple threads to concurrently send messages, the message order is determined based on the order in which the messages are received by the Message Queue for Apache RocketMQ broker. This order may be different from the sending order on the business side.
TCP SDK sample code
For more information about the TCP sample code, see Send and subscribe to ordered messages by using the Java SDK.
Upgrade of existing topics
If you use ordered messages 1.0 and want to upgrade ordered messages 1.0 to ordered messages 2.0, submit a ticket to contact the product team. The product team can help you migrate your business.
- Can a single message be an ordered message, a scheduled message, and a transactional
message at the same time?
No, ordered messages, scheduled messages, and transactional messages are different and mutually exclusive message types.
- In which regions can I use ordered messages?
Ordered messages are supported in all Alibaba Cloud regions and Finance Cloud regions in which Message Queue for Apache RocketMQ is available.
- Why is the performance of globally ordered messages not high?
Globally ordered messages are processed in the FIFO order. If a message is not consumed, the next message remains in the topic queue and cannot be processed. To improve the transactions per second (TPS) of globally ordered messages, upgrade your instance configuration and make sure that the application on the message client processes local business logic in a timely manner.
- What transmission modes do ordered messages support?
Ordered messages support only the reliable synchronous transmission mode that ensures strict message order. Asynchronous transmission is not supported.
- Do ordered messages support clustering consumption and broadcasting consumption?
Ordered messages support clustering consumption but do not support broadcasting consumption.