Ordered messages are delivered and consumed in the same order. A producer sends messages in an order, and a consumer subscribes to and consumes these messages from the earliest received message. Ordered messages are often used in business (such as financial securities and e-commerce) that need to consume messages in the order that the messages are produced. This topic describes the concepts, scenarios, working mechanism, and usage notes of Ordered Message 1.0 in Message Queue for Apache RocketMQ.
What are ordered messages?
In Message Queue for Apache RocketMQ, ordered messages are messages that are sent and consumed in a strict order. Messages under a specified topic are published and consumed in a strict first-in-first-out (FIFO) order. That is, the earliest published message is consumed first.
Partitionally ordered messages
All messages under a specified topic are partitioned by Sharding Key. The messages in each partition are published and consumed in a strict FIFO order. There is no requirement on the order of messages from different partitions.
- Sharding Key: key field used by ordered messages to distinguish partitions. Message Queue for Apache RocketMQ routes messages with the same Sharding Key to a partition. The earliest published message in the partition is consumed first. This key field is different from that of normal messages.
- Partition: topic-level partition. Each topic contains one or more partitions. Messages under the same topic are distributed to these partitions. Topic-level partitions are referred to as logical partitions in this topic. For more information, see Terms.
- Physical partition: physical unit for storing messages. Each physical partition resides on a specified node of a machine.
Partitionally ordered messages are used in scenarios where the performance requirements are high and messages in each partition must be published and consumed in a strict FIFO order.
- For example, if a verification code is required for user registration and the user ID is used as the Sharding Key, messages sent by the same user are published and consumed in an FIFO order.
- In e-commerce scenarios, the order ID is used as the Sharding Key. Then, messages about the creation, payment, delivery, and refund status of the same order are published and consumed in an FIFO order.
All internal e-commerce systems in Alibaba Group use partitionally ordered messages to ensure the service order and performance.
Globally ordered messages
All messages under a topic are published and consumed in a strict FIFO order.
Globally ordered messages are typically used in scenarios where the performance requirements are not high and all messages must be published and consumed in a strict FIFO order.
For example, in securities processing scenarios where the topic is RMB to USD exchange, the transactions with the same bid price are processed in the FIFO order. In this scenario, the system must publish and consume messages under the topic in a global FIFO order.
How do ordered messages work?Globally ordered messages work in the same way as partitionally ordered messages. The following section describes how to ensure that messages are sent and consumed in the same order in Message Queue for Apache RocketMQ. Partitionally ordered messages are used in the example.
- Message sending
In the preceding figure, messages for orders A and B are produced in the sequence of A1, B1, A2, A3, B2, and B3. The messages for one order must be sent and consumed in the same sequence as these messages are produced. For example, the messages for Order A are sent and consumed in the sequence of A1, A2, and A3. Normal messages for Order A may be sent to different queues after polling, and the original message order cannot be retained. In comparison, ordered messages with the same Sharding Key (such as the same order ID) are sequentially routed to a partition in Message Queue for Apache RocketMQ.
In general, messages are produced and sent in the same time sequence. For example, if messages about order creation, order payment, and payment success are produced in sequence, these messages must be sent in this sequence. The producer needs to ensure that messages are produced and sent in the same sequence. We recommend that you configure the producer to send messages one by one. Messages sent in parallel may arrive in the queue out of order.
- Message storage
In the preceding figure of Ordered Message 1.0, each logical partition corresponds to a physical partition. When messages are sent to the logical partitions in order, these messages are stored in the corresponding physical partitions in the same order.
- Message consumption
Message Queue for Apache RocketMQ delivers messages to a consumer in the order in which messages are stored. The consumer then consumes the messages from the earliest received message.
The consumer consumes messages of the same Sharding Key one by one. This ensures that the earliest stored message, namely the earliest published message, is consumed first.
In Ordered Message 1.0, a logical partition corresponds to only one physical partition, and the preset number of partitions cannot be modified. If the service traffic grows, a physical node may not be able to meet the capacity requirement. In this case, neither scaling nor data splitting can be performed. Scaling affects the policy and sequence of message routing, while data splitting requires additional reconstruction costs. Therefore, Ordered Message 1.0 applies to scenarios where the service scale is stable, message volume fluctuates slightly, and no future scaling is required.
If scaling may be performed to meet the requirement of traffic growth, we recommend that you use Ordered Message 2.0, which supports unlimited scale-out without negative impacts on services. Ordered Message 2.0 avoids the issue of unbalanced message distribution among partitions, and enables a smooth service switchover in seconds when a physical node fails. For more information, see Ordered Message 2.0.
When you use Ordered Message 1.0, take note of the following items:
- One group ID corresponds to one topic type. One group ID cannot be used to send and receive both ordered messages and unordered messages.
- We recommend that you avoid congestion of globally ordered messages. You can run multiple instances simultaneously to ensure service continuity. If the active instance fails, another instance can immediately take over the services. Only one instance is active at a time.
- Messages in a partition must be sent and subscribed to one by one. Otherwise, the messages may become out of order.
- Can a single message be an ordered message, a scheduled message, and a transactional
message at the same time?
No. Ordered messages, scheduled messages, and transactional messages are different and mutually exclusive.
- In which regions can I use ordered messages?
Ordered messages are supported in all Alibaba Cloud regions and AntCloud regions in which Message Queue for Apache RocketMQ is available.
- Why is the performance of globally ordered messages unsatisfactory?
Globally ordered messages are processed in a strict FIFO order. If a message is not consumed, the next message will remain in the topic queue and cannot be processed. To improve the performance of globally ordered messages, upgrade your instance configuration and ensure that the application on the message client processes local service logic as quickly as possible.
- What methods can be used to send ordered messages?
Ordered messages can be transmitted only in the synchronous mode, but not the asynchronous mode, which ensures the strict message order.
- Do ordered messages support clustering consumption and broadcasting consumption?
Ordered messages support clustering consumption but do not support broadcasting consumption.
TCP SDK sample code
HTTP SDK sample code
- Java: Send and subscribe to ordered messages
- Go: Send and subscribe to ordered messages
- Python: Send and subscribe to ordered messages
- Node.js: Send and subscribe to ordered messages
- PHP: Send and subscribe to ordered messages
- C#: Send and subscribe to ordered messages
- C++: Send and subscribe to ordered messages