How to use consumer groups in RocketMQ - ApsaraMQ for RocketMQ

This topic describes the definition, model, internal attributes, behavioral constraints, version compatibility, and usage suggestions for consumer groups in ApsaraMQ for RocketMQ.

Definition

A consumer group is a load balancing group in ApsaraMQ for RocketMQ that consists of multiple consumers with identical consumption behaviors.

Unlike consumers, which are running entities, consumer groups are logical resources. In ApsaraMQ for RocketMQ, you can initialize multiple consumers in a consumer group to scale out consumption performance and implement high-availability disaster recovery.

The following consumption behaviors are defined for a consumer group. All consumers in the same group consume messages based on these shared behaviors and the group's load balancing policy.

Subscription relationship: ApsaraMQ for RocketMQ manages and tracks subscription relationships at the consumer group level. For more information, see Subscription relationships.
Delivery order: The ApsaraMQ for RocketMQ server supports both ordered and concurrent delivery of messages to consumers. The delivery method is configured at the consumer group level. For more information, see Ordered messages.
Consumption retry policy: This is the policy that applies when a consumer fails to consume a message. The policy includes the maximum number of retries and dead-letter queue settings. For more information, see Consumption retries.

Model relationship

In the ApsaraMQ for RocketMQ domain model, a consumer group is positioned in the message flow as follows: 消费组

Producers produce and send messages to the ApsaraMQ for RocketMQ broker.
The ApsaraMQ for RocketMQ broker stores the messages in the queue that is specified by the topic in the order in which the messages are received.
Consumers obtain and consume messages from the ApsaraMQ for RocketMQ broker based on the specified subscriptions.

Internal attributes

Consumer group name

Definition: The name of a consumer group. It is used to distinguish the group from others and must be globally unique within a cluster.
Value: A consumer group is created and configured by the user. For more information about the naming conventions, see Parameter limits.

Consumption pattern

Definition: The pattern in which consumers consume messages. ApsaraMQ for RocketMQ provides two consumption patterns for different scenarios: CLUSTERING and LITE_SELECTIVE. For more information, see Lightweight topic model.
Value: The default consumption pattern is CLUSTERING.

Delivery order

Definition: The order in which ApsaraMQ for RocketMQ delivers messages to a consumer client.
ApsaraMQ for RocketMQ supports ordered delivery and concurrent delivery for different consumption scenarios. To prevent potential exceptions during message processing, the delivery order cannot be changed after a consumer group is created. For more information, see Ordered messages.
Value: The default delivery method is concurrent delivery. If the consumption pattern is LITE_SELECTIVE, the delivery method must be ordered delivery.

Consumption retry policy

Definition: The policy that the system uses to retry message delivery when a consumer fails to consume a message. Based on this policy, the system redelivers the message to the consumer for another consumption attempt. For more information, see Consumption retries.
Value:
A consumption retry policy includes:
- Maximum retries: The maximum number of times that a message can be redelivered. If a message is not consumed after the maximum number of retries is reached, the message is delivered to a dead-letter queue or discarded.
- Retry interval: The interval at which the ApsaraMQ for RocketMQ server redelivers a message.
For more information about the value ranges and default values of the maximum number of retries and the retry interval, see Parameter limits.
Constraint: The retry interval is valid only for PushConsumers.

Subscription relationshipDefinition: The set of subscription relationships associated with the consumer group. This includes the topics that consumers subscribe to and the message filtering rules. Consumers dynamically register subscription relationships with a consumer group. The ApsaraMQ for RocketMQ server persists these subscription relationships and uses them to track message consumption progress. For more information, see Subscription relationships.

Limits on consumption logic

In the domain model of ApsaraMQ for RocketMQ, consumers are managed in consumer groups. Consumers in the same consumer group share all messages that are delivered to the group. To ensure that messages are loaded and consumed as expected in a consumer group in ApsaraMQ for RocketMQ, the following consumption logic must be consistent for all consumers in the group:

Message delivery order
Consumption retry policy

Version compatibility

The message delivery order and consumption retry policy must be consistent for all consumers in a consumer group. The following items describe how the consistency is ensured in different ApsaraMQ forRocketMQ broker versions:

ApsaraMQ for RocketMQ 5.x brokers: Consumers obtain the message delivery order and consumption retry policy from the consumer group with which the consumers are associated. The consumption logic is the same for all consumers. You do not need to specify the message delivery order or consumption retry policy on the client.
ApsaraMQ for RocketMQ 3.x and 4.x brokers: The message delivery order and consumption retry policy are defined by API operations on your client. You must specify the message delivery order and consumption retry policy on the client to ensure that the consumption logic of all consumers in the consumer group is the same.

If you use an SDK of an earlier version to access a ApsaraMQ for RocketMQ 5.x broker, the consumption logic of consumers is determined by the configurations of API operations on the consumer client.

Usage suggestions

Split into groups based on business logic

In ApsaraMQ for RocketMQ, consumers and topics have a many-to-many relationship. Follow these principles when you design and create consumer groups:

Ensure a consistent delivery order: All consumers in the same consumer group must use the same delivery order, which can be either ordered or concurrent. Do not use the same consumer group for business scenarios that require different delivery orders.
Group by business type: Typically, one consumer group corresponds to one topic. Different business domains have different message consumption requirements, such as different message filtering attributes and consumption retry policies. Therefore, you should use different consumer groups to consume messages from topics that belong to different business domains. We recommend that a single consumer group consumes messages from no more than 10 topics.

Avoid using automated mechanisms to manage consumer groups

In the ApsaraMQ for RocketMQ architecture, consumer groups are logical resources used for state management. Each consumer group is associated with consumption states, message backlogs, observable metrics, and monitoring data. Therefore, you must strictly manage consumer group resources in a production environment. Do not arbitrarily add, delete, modify, or query consumer groups.