All Products
Search
Document Center

ApsaraMQ for RocketMQ:Clustering consumption and broadcasting consumption

Last Updated:Dec 20, 2023

ApsaraMQ for RocketMQ supports clustering consumption and broadcasting consumption. You can set the consumption mode of consumers in a cluster to clustering or broadcasting on clients to specify whether a message is consumed by only one consumer or all consumers in the cluster.

Background information

ApsaraMQ for RocketMQ is a messaging system that is based on the Publish-Subscribe pattern. In ApsaraMQ for RocketMQ, a consumer subscribes to topics to obtain and consume messages.

Multiple consumers that are identified by the same group ID belong to the same consumer cluster and must use the same consumption logic and tags. For more information, see Subscription consistency.

Clustering consumption

Basic concept

In clustering consumption mode, multiple consumers in a group consume messages from a specific topic. ApsaraMQ for RocketMQ delivers messages to different consumers and each message is consumed by only one consumer.

图片2.png

Scenarios

  • Message load balancing: If a large number of messages are processed in your business, you can use the clustering consumption mode to distribute messages to multiple consumers to implement load balancing. This way, each consumer processes only a small number of messages. This helps improve message processing efficiency and throughput.

  • Distributed data processing: For scenarios in which a large amount of data needs to be processed, you can use the clustering consumption mode to distribute data to multiple nodes. This way, each node processes only a small amount of data. This improves the processing rate.

Usage notes

  • In clustering consumption mode, a failed message is not necessarily redelivered to the same machine.

  • Make sure that you specify the same consumption mode for consumers in the same group. If you specify different consumption modes for consumers in the same group, the broadcasting consumption mode takes effect.

    For example, you create a group that consists of five consumers. If you specify the clustering consumption mode for two consumers in the group and the broadcasting consumption mode for the other three consumers in the group, the broadcasting consumption mode is used for all five consumers.

Broadcasting consumption

Basic concept

In broadcasting consumption mode, when multiple consumers in a group consume messages in a specific topic, ApsaraMQ for RocketMQ delivers each message to all consumers in the cluster. This ensures that each message is consumed by each consumer at least once.

This mode is suitable for scenarios in which multiple consumers concurrently process the same group of messages, such as order processing.

图片3.png

Scenarios

  • Notifications and announcements: When an important announcement or notification is released, the system can use the broadcasting consumption mode to send the notification or announcement to all users who subscribe to the message. This ensures that each user receives the notification or announcement in a timely manner.

  • Real-time data synchronization: In some cases, you want to synchronize specific data in a distributed system in real time to maintain data consistency among multiple nodes. You can use the broadcasting consumption mode to send the notification about the data change to all nodes to which you want to synchronize the data change. This way, data on each node can be consistent.

Usage notes

  • In broadcasting consumption mode, ApsaraMQ for RocketMQ ensures that a message is consumed by consumers at least once, but does not redeliver messages that fail to be consumed. You need to manually handle consumption failures.

  • In broadcasting consumption mode, a consumer starts message consumption from the latest message each time the consumer is restarted. A consumer automatically skips the messages that are sent to the ApsaraMQ for RocketMQ broker when the consumer is stopped. Proceed with caution when you use this mode.

  • In broadcasting consumption mode, each message is repeatedly processed by a large number of consumers. We recommend that you use the clustering consumption mode in most cases.

  • Make sure that you specify the same consumption mode for consumers in the same group. If you specify different consumption modes for consumers in the same group, the broadcasting consumption mode takes effect.

    For example, you create a group that consists of five consumers. If you specify the clustering consumption mode for two consumers in the group and the broadcasting consumption mode for the other three consumers in the group, the broadcasting consumption mode is used for all five consumers.

Differences between clustering consumption and broadcasting consumption

The availability of features varies based on the consumption mode. The following table lists the features that are available in clustering consumption mode and the features that are available in broadcasting consumption mode.

Feature

Clustering consumption mode

Broadcasting consumption mode

TCP client SDKs

对号

对号

HTTP client SDKs

对号

错号

Ordered messages

对号

错号

Consumer offset resetting

对号

错号

Message retry

对号

错号

Query and alerting of accumulated messages

对号

错号

Subscription query

对号

错号

Consumer progress

Maintained on ApsaraMQ for RocketMQ brokers.

  • The service reliability is higher in clustering consumption mode than that in broadcasting consumption mode. After a consumer is restarted, the consumer continues to consume messages based on the last consumption progress.

  • Message retry is supported by ApsaraMQ for RocketMQ brokers. For more information, see Message retry.

Maintained on clients.

The probability that a message is repeatedly consumed in broadcasting consumption mode is higher than that in clustering consumption mode. Each time a consumer is restarted, the consumer starts consumption from the latest message.

Configure the clustering consumption mode and the broadcasting consumption mode

ApsaraMQ for RocketMQ allows you to change consumption modes on clients. To change the consumption mode, you must configure the corresponding parameter in the code of the SDK that subscribes to messages. If you do not configure the parameter, the clustering consumption mode is used.

TCP client SDKs

Sample code for the SDK for Java

  • Clustering consumption

    // Set the MessageModel parameter to CLUSTERING. If you do not configure this parameter, the clustering consumption mode is automatically used. 
    properties.put(PropertyKeyConst.MessageModel, PropertyValueConst.CLUSTERING);
  • Broadcasting consumption

    // Set the MessageModel parameter to BROADCASTING. 
    properties.put(PropertyKeyConst.MessageModel, PropertyValueConst.BROADCASTING); 

Sample code for the SDK for C or C++

  • Clustering consumption

    // Set the MessageModel parameter to CLUSTERING. If you do not configure this parameter, the clustering consumption mode is automatically used. 
    factoryInfo.setFactoryProperty(ONSFactoryProperty:: MessageModel, ONSFactoryProperty::CLUSTERING);
  • Broadcasting consumption

    // Set the MessageModel parameter to BROADCASTING. 
    factoryInfo.setFactoryProperty(ONSFactoryProperty:: MessageModel, ONSFactoryProperty::BROADCASTING);

Sample code for the SDK for .NET

  • Clustering consumption

    // Set the MessageModel parameter to CLUSTERING. If you do not configure this parameter, the cluster consumption mode is automatically used. 
    factoryInfo.setFactoryProperty(ONSFactoryProperty.MessageModel, ONSFactoryProperty.CLUSTERING); 
  • Broadcasting consumption

    // Set the MessageModel parameter to BROADCASTING. 
    factoryInfo.setFactoryProperty(ONSFactoryProperty.MessageModel, ONSFactoryProperty.BROADCASTING);     

HTTP client SDKs

By default, HTTP client SDKs support the clustering consumption mode. You do not need to configure a parameter to specify the consumption mode.

FAQ

Why does the clustering consumption mode not take effect after I specify it?

The issue may occur due to one of the following reasons:

  • The messages that you process are ordered messages. Ordered messages support only clustering consumption.

  • The SDK that you use is of the HTTP client type. HTTP clients support only clustering consumption.

Can I specify different consumption modes for different consumers in a group?

No, you cannot specify different consumption modes for different consumers in a group.

Make sure that you specify the same consumption mode for consumers in the same group. If you specify different consumption modes for consumers in the same group, the broadcasting consumption mode takes effect.

References

For information about the complete sample code for messaging, see Overview.