Send Large Message

Last Updated: Apr 18, 2017

Context

An Alibaba Cloud Message Service (MNS) queue has a maximum message size of 64 KB. This is about enough to meet the needs of messages used as a control-flow information exchange channel under normal circumstances. However, in some special scenarios where messages are bigger in size than that, message slices will be used.

So how can we transmit messages larger than 64 KB with MNS without using message slices? There is a solution.

Solution

  1. If a message is found to be larger than 64 KB, the producer first uploads it to OSS before sending it to MNS;

  2. Then, the producer sends to MNS the appropriate object information for the message data;

  3. When reading a message from the MNS queue, the consumer determines whether the message is OSS object information;

  4. If the message is OSS object information, then the consumer can download the appropriate object information from OSS, and return it as a message to the program at the upper layer.

  5. The messages smaller than 64 KB are still transmitted directly using MNS.

The specific process is shown below:

Large Message

# Program implementation

Sample code for large messages is the Java implementation of the solution above. Its main functionalities are encapsulated into a class, namely BigMessageSizeQueue

BigMessageSizeQueue provides the following public method:

  1. public BigMessageSizeQueue(CloudQueue cq, OSSClient ossClient, String ossBucketName)
  2. //Constructor, cq is a common mnsqueue object; ossClient and ossBucketName include OSS region and bucket information needed for relaying large messages
  3. public Message putMessage(Message message) // Send a message
  4. public Message popMessage(int waitSeconds) // Receive a message
  5. public void deleteMessage(String receiptHandle) // Delete a message
  6. public void setBigMessageSize(long bigMessageSize) // Set a large message threshold (a message larger than this threshold will be transmitted through OSS), which by default is 64 KB;
  7. public void setNeedDeleteMessageObjectOnOSSFlag(boolean flag) // Choose whether to delete messages in OSS, which is set to Yes by default;

For details about the sample code, see the code in Demo.java attached.

NOTE

  1. Large messages consume a considerable amount of network bandwidth. Therefore, when using this solution to send large messages, the network bandwidth for the producer and consumer may be a bottleneck.

  2. Transmitting large messages takes longer time and is more susceptible to network fluctuations. Thus, necessary retries are recommended at the upper layer.

Thank you! We've received your feedback.