edit-icon download-icon

Strict FIFO Queue

Last Updated: Apr 18, 2017

Context

The queue provided by Alibaba Cloud Message Service features high reliability, high availability, and high concurrency. Data of each queue is persistently written with three copies in Alibaba Cloud Apsara distributed platform. Each queue contains at least two servers providing services, and each server supports highly-concurrent access. With such distributed characteristics, the Message Service queue cannot ensure strict first-in-first-out (FIFO) of messages as the traditional single-host queue, and can only be basically ordered.

If there are multiple message senders in a queue, strict message ordering is meaningless due to concurrency and network delay, as the actual sending order of messages from multiple senders and the real order in which messages arrive at the server are unknown. Likewise, when multiple receivers concurrently receive messages, the real message processing order is unknown.

Therefore, the message ordering is meaningful when there is only one sender (a thread or multiple threads) and one receiver; and the actual sending and receiving orders of messages can be aware and recorded.

Solution

The following solution is designed based on the above assumptions to meet users’ requirements on the message order and ensure that messages are received and consumed in an order they are sent.

Procedure:

  1. The sender colors messages and adds SeqID (for example, #num#) to the messages.

  2. The receiver restores the messages, sorts them by SeqID, and returns the messages to the upper layer. Backend threads are provided for messages that have been received to ensure that the messages will not be repeatedly consumed.

  3. To avoid SeqID loss due to sender or receiver failure, SeqID will be persistently stored in a local disk file. (SeqID can be saved to other storage devices or databases, such as OSS, Table Store, or ApsaraDB for RDS.)

FIFO Queue

Program description

Attachment is a solution in Python version (relying on the Message Service Python SDK). In which, oredered_queue.py files in OrderedQueueWrapper type are provided to wrap common Message Service queues to ordered queues.

OrderedQueueWrapper provides the SendMessageInOrder() and ReceiveMessageInOrder() methods. SendMessageInOrder() is used to color messages, while ReceiveMessageInOrder() is used to restore messages, and return them to the receiver in order.

In addition, send_message_in_order.py and receive_message_in_order.py provide program samples for the sender and receiver to use OrderedQueueWrapper.

  1. send_message_in_order.py
  2. #init orderedQueue
  3. seqIdConfig = {"localFileName":"/tmp/mns_send_message_seq_id"} # Specifies the disk file to persistently send the SeqID.
  4. seqIdPS = LocalDiskStorage(seqIdConfig)
  5. orderedQueue = OrderedQueueWrapper(myQueue, sendSeqIdPersistStorage = seqIdPS)
  6. orderedQueue.SendMessageInOrder(message)
  7. receive_message_in_order.py
  8. #init orderedQueue
  9. seqIdConfig = {"localFileName":"/tmp/mns_receive_message_seq_id"} #Specifies the disk file to persistently receive the SeqID.
  10. seqIdPS = LocalDiskStorage(seqIdConfig)
  11. orderedQueue = OrderedQueueWrapper(myQueue, receiveSeqIdPersistStorage = seqIdPS)
  12. recv_msg = orderedQueue.ReceiveMessageInOrder(wait_seconds)

Running method:

  1. Configure g_endpoint, g_accessKeyId, g_accessKeySecret, and g_testQueueName in send_message_in_order.py and receive_message_in_order.py.

  2. Run send_message_in_order.py.

  3. Run receive_message_in_order.py (you can perform this step without completion of step 2.)The sender sends 20 messages, and the receiver receives the 20 messages in order.

Screenshot 1

You can also run the test case of oredered_queue.py to compare the ordered queue with the common Message Service queue (the endpoint and AK need to be configured):

Command: $python oredered_queue.py

Non-strictly ordered: (Entire messages are ordered while some adjacent messages are disordered. This also indicates that multiple servers in a single queue of Message Service simultaneously provide services.)

Screenshot 2

Strictly ordered:

Screenshot 3

NOTE:

  1. This post mainly aims to provide message ordering solutions. Code in this post has not been strictly tested. You are not advised to directly use the code in production environments without test. Meanwhile, the program is complete in a rush, which may contain errors. Looking forward to your correction.

  2. In normal conditions, the SeqIDs of the sender and receiver should match those of the messages (colored) in the queue. When the queue is deleted for recreation, ensure that the SeqID in the disk file is consistent with that in the queue. You are not advised to send a non-colored message to a queue with colored messages.

  3. A short message validation period in the queue and the actual processing result of each message may affect the message order. You need to process such disordered messages in your program.

Download the sample program:

Python sample code for an ordered queue

Thank you! We've received your feedback.