This topic introduces concepts of Message Queue for Apache RocketMQ clustering consumption and broadcasting consumption, their application scenarios, and precautions for use.

Message Queue for Apache RocketMQ is a messaging system that is based on the message publish/subscribe pattern. In Message Queue for Apache RocketMQ, consumers subscribe to a topic to obtain and consume messages. Consumer applications are deployed in clusters in a distributed system. Therefore, Message Queue for Apache RocketMQ defines the following terms:

  • Cluster: Consumers using the same group ID belong to the same cluster. These consumers must have the same consumption logic, including the same tags. For more information, see Subscription consistency.
  • Clustering consumption: In this pattern, a message only needs to be processed by any consumer in the cluster for Message Queue for Apache RocketMQ.
  • Broadcasting consumption: In this mode, Message Queue for Apache RocketMQ broadcasts each message to all clients registered in the cluster to ensure that the message is consumed by each consumer at least once.

Clustering consumption pattern

  • Scenarios
    This pattern is applicable to the scenario where consumer instances are deployed in a cluster and each message needs to be processed only once. The consumption progress is maintained on the Message Queue for Apache RocketMQ broker, ensuring high reliability. The following figure shows a consumption example.Clustering consumption
  • Precautions
    • In clustering consumption pattern, each message is delivered to only one machine in the cluster for processing. If a message needs to be processed by each machine in the cluster, use the broadcasting consumption pattern instead.
    • In clustering consumption pattern, a message may be routed to different machines during consumption retries.

Broadcasting consumption pattern

  • Scenarios
    This pattern is applicable to the scenario where consumer instances are deployed in a cluster and each message needs to be processed by each consumer in the cluster. The following figure shows a consumption example.Broadcasting consumption
  • Precautions
    • Ordered messages are not supported in broadcasting consumption pattern.
    • Consumption offsets cannot be reset in broadcasting consumption pattern.
    • Each message needs to be processed by multiple machines with the same subscription logic.
    • The consumption progress is maintained at the client. The ratio of consumption repetition is higher than that of the clustering consumption pattern.
    • In broadcasting consumption pattern, Message Queue for Apache RocketMQ ensures that each message is consumed by each client at least once, but does not resend any messages that fail to be consumed. Therefore, your customer needs to pay attention to consumption failures.
    • In broadcasting consumption pattern, the consumption starts from the latest message every time the client is restarted, but skips the messages sent to the Message Queue for Apache RocketMQ broker when the client is stopped. For this reason, use this pattern with caution.
    • In broadcasting consumption pattern, each message is repeatedly processed by many clients. Therefore, we recommend that you use the clustering consumption pattern whenever possible.
    • In broadcasting consumption mode, the Message Queue for Apache RocketMQ broker does not maintain the consumption progress, so you cannot query message accumulation, set message accumulation alerts, or query subscriptions in the Message Queue for Apache RocketMQ console.

Use the clustering consumption pattern to simulate the broadcasting consumption pattern

  • Scenarios
    This is applicable to scenarios where each message needs to be processed by multiple machines with the same or different logic. The following figure shows a consumption example.Use the clustering consumption pattern to simulate the broadcasting consumption pattern

    If the broadcasting consumption pattern is required for your business, create multiple group IDs to subscribe to the same topic.

  • Precautions
    • The consumption progress is maintained on the broker, and the reliability is higher than that in broadcasting consumption pattern.
    • For one group ID, one or more consumer instances can be deployed. When multiple consumer instances are deployed, these instances form a cluster to work together for message consumption. Assume that three consumer instances C1, C2, and C3 are deployed for group ID 1. These instances share the messages sent from the Message Queue for Apache RocketMQ broker to group ID 1. In addition, these instances must subscribe to same topics with same tags. For more information, see Subscription consistency.

References

For more information about how to configure the clustering consumption pattern and broadcasting consumption pattern, see the following documents: