edit-icon download-icon

Clustering and broadcasting consumption

Last Updated: Sep 18, 2018

This topic introduces the concept of MQ clustering consumption and broadcasting consumption, their scenarios, and usage precautions.

Basic concept

MQ is based on a publishing and subscription (Pub/Sub) model. Consumers subscribe to a topic to obtain and consume messages. As the subscribers are usually distributed systems that consist of multiple machines deployed in a cluster, MQ defines the following terms:

Cluster: Consumers using the same consumer ID belong to the same cluster. These consumers have identical consumption logic (including Tag usage) and can be considered as one consumption node logically.

Clustering consumption: In this model, only one message needs to be processed by any consumer in the cluster.

Broadcasting consumption: In this model, MQ broadcasts each message to all clients registered in the cluster to ensure that the message is consumed by each machine at least once.

Scenario Comparison

Clustering consumption mode:

clustering consumption

Scenarios and usage instructions:

  • The consumption end has a cluster deployment, and each message only needs to be processed once.
  • The consumption progress is maintained on the server, which leads to higher reliability.
  • In clustering consumption mode, each message will be delivered to only one machine for processing. If a message needs to be processed by every machine in the cluster, broadcast mode should be used.
  • In clustering consumption mode, there is no guarantee that the re-delivery of a failed message will be routed to the same machine, so no definitive assumptions should be made when processing messages.

Broadcasting consumption mode:

broadcasting consumption

Scenarios and usage instructions:

  • Ordered message is not supported in broadcasting consumption mode.
  • Each message needs to be processed by multiple machines with the same logic.
  • Consumption progress is maintained on the client side, so the probability of repeated messages is slightly higher than in clustering consumption mode.
  • In broadcast mode, MQ ensures that each message will be consumed at least once by each client, but the message which failed to be consumed will not be re-delivered after the failure, so the service provider needs to pay attention to consumption failure.
  • In broadcast mode, the latest message will be consumed by default when the consumer is started for the first time. The consumption progress is stored persistently in a local hidden file on the client. It is not suggested to delete this file, otherwise some messages will be lost.
  • In broadcast mode, each message will be repeatedly processed by multiple clients. It is recommended to use cluster mode as much as possible.
  • Currently only Java client is supported in broadcast mode.
  • In broadcast mode, the server does not maintain the consumption progress, so message accumulation queries or accumulation alarms are not supported in the MQ console.

Use cluster mode to simulate broadcast:

If broadcast mode is needed, you can create multiple consumer IDs to subscribe to the same topic.

cluster simulate broadcast

Scenarios and usage instructions:

  • Each message needs to be processed by multiple machines, and each machine can have the same or different logic.
  • The consumption progress is maintained on the server, so there is higher reliability than in broadcast mode.
  • For one consumer ID, one or more consumer instances can be deployed. When multiple consumer instances are deployed, the instances constitute a cluster (share message consumption together). Assuming that three consumer instances C1, C2, and C3 are deployed for consumer ID1, these instances will share the messages that the server sends to consumer ID1. These instances should subscribe to the same topics with the same tags.
Thank you! We've received your feedback.