All Products
Search
Document Center

Clustering consumption and broadcasting consumption

Last Updated: Sep 13, 2019

This topic introduces concepts of Alibaba MQ clustering consumption and broadcasting consumption, their scenarios, and precautions for use.

Concepts

Alibaba MQ is a messaging system that is based on message publishing and subscription. Consumers subscribe to a topic to retrieve and consume messages. As the subscribers are usually distributed systems that consist of multiple machines deployed in a cluster, Alibaba MQ defines the following terms:

Cluster: Consumers using the same group ID belong to the same cluster. These consumers have identical consumption logic (including tag usage) and can be considered logically as one consumption node.

Clustering consumption: In this model, any message only needs to be processed by any consumer in the cluster.

Broadcasting consumption: In this model, Alibaba MQ broadcasts each message to all clients registered in the cluster to ensure that the message is consumed by each machine at least once.

Scenario comparison

Clustering consumption mode:

clusteringconsumption

Scenarios and usage instructions

  • Consumer instances are deployed in a cluster and each message needs to be processed only once.
  • The consumption progress is maintained on the broker, so the reliability is high.
  • In clustering consumption mode, each message is delivered to only one machine in the cluster for processing. If a message needs to be processed by every machine in the cluster, use the broadcasting consumption mode.
  • In clustering consumption mode, there is no guarantee that the re-delivery of a failed message will be routed to the same machine, so no definitive assumptions should be made when processing messages.

Broadcasting consumption mode:

broadcastingconsumption2

Scenarios and usage instructions

  • Ordered messages are not supported in broadcasting consumption mode.
  • Resetting consumption offsets is not supported in broadcasting consumption mode.
  • Each message needs to be processed by multiple machines with the same logic.
  • The consumption progress is maintained at the client. The ratio of repetition is higher than that of the clustering consumption mode.
  • In broadcasting consumption mode, Alibaba MQ ensures that each message is consumed by each client at least once, but does not resend messages that fail to be consumed. Therefore, the business side needs to pay attention to consumption failures.
  • In broadcasting consumption mode, the consumption starts from the latest message by default when the consumer is started everytime, skipping the messages sent to the Alibaba MQ server when the consumer is stopped. Consider thouroughly before using this mode.
  • In broadcasting consumption mode, each message is processed repeatedly by many clients. Therefore, the clustering consumption mode is recommended.
  • Currently, only Java clients support the broadcasting consumption mode.
  • In broadcasting consumption mode, the broker does not maintain the consumption progress, so you cannot query message accumulation, set message accumulation alarms, or query subscription in the Alibaba MQ console.

Use the clustering consumption mode to simulate the broadcasting consumption mode:

If the broadcasting consumption mode is needed for your business, you can create multiple group IDs to subscribe to the same topic.

cd+bd

Scenarios and usage instructions

  • Each message needs to be processed by multiple machines and the logic of each machine can be either the same or different.
  • The consumption progress is maintained on the broker, and the reliability is higher than that in the broadcasting consumption mode.
  • For one group ID, one or more consumer instances can be deployed. When multiple consumer instances are deployed, the instances form a cluster to work together for message consumption. Assume that three consumer instances C1, C2, and C3 are deployed for group ID 1. The three instances share the messages sent from the broker to group ID 1. These instances must subscribe to the same topics with the same tags.