ApsaraMQ for RocketMQ:Consumer load balancing

How it works

The broker distributes individual messages from a topic evenly across all consumers in a consumer group, regardless of which queue the messages belong to. Multiple consumers can process messages from the same queue concurrently.

In this example, Consumer Group A has three consumers: A1, A2, and A3. All three consume messages from Queue1. The broker assigns each message to one consumer at a time.

When a consumer receives a message, the broker locks that message so it becomes invisible to other consumers. The message stays locked until the consumer acknowledges it or the lock times out. This prevents duplicate processing under normal conditions.

Ordered message handling

For ordered messages, ApsaraMQ for RocketMQ guarantees that messages within the same message group are processed in the exact order they were stored on the broker.

Consider four ordered messages (M1 through M4) in message group G1 of Queue1. If consumer A1 is processing M1 and M2, consumer A2 cannot start processing M3 or M4 until A1 has submitted the consumption status for M1 and M2. The broker enforces strict sequential locking to maintain order.

Advantages over queue-based load balancing

When to use

Message-based load balancing is suitable for most online event processing workloads where each message is handled independently -- for example, order processing, notification dispatch, and real-time event handling.

For stream processing or aggregation workloads that require batching messages from the same queue, use queue-based load balancing instead.

Scope

Message-based load balancing is the only policy available for Push Consumer and Simple Consumer types on broker version 5.x. It is enabled by default and requires no additional configuration.

Example

Push Consumer and Simple Consumer types use message-based load balancing automatically. The following Java examples show both consumer types processing messages without any load balancing configuration.

Push Consumer with a message listener:

// Push Consumer: implement a message listener to process messages.
// Load balancing is handled automatically by the broker.
MessageListener messageListener = new MessageListener() {
    @Override
    public ConsumeResult consume(MessageView messageView) {
        System.out.println(messageView);
        // Return the consumption result.
        return ConsumeResult.SUCCESS;
    }
};

Simple Consumer with manual acknowledgment:

// Simple Consumer: pull messages, process them, and acknowledge each one.
// Load balancing is handled automatically by the broker.
try {
    List<MessageView> messageViewList = simpleConsumer.receive(10, Duration.ofSeconds(30));
    messageViewList.forEach(messageView -> {
        System.out.println(messageView);
        try {
            // Acknowledge the message after processing.
            simpleConsumer.ack(messageView);
        } catch (ClientException e) {
            e.printStackTrace();
        }
    });
} catch (ClientException e) {
    // If the pull fails due to throttling or other issues, retry the request.
    e.printStackTrace();
}

How it works

The broker assigns each queue in a topic to exactly one consumer in the consumer group. Each consumer then processes all messages from its assigned queues.

In this example, a topic has three queues (Queue1, Queue2, Queue3) and the consumer group has two consumers. Because each queue is assigned to one consumer, consumer A2 gets two queues while A1 gets one. If there are fewer queues than consumers, some consumers receive no queues and remain idle.

Each consumer follows this processing sequence: pull messages from the assigned queue, submit the consumption offset, and persist the offset. Because the consumption status is not returned to the queue when a consumer pulls messages, each queue must be exclusively assigned to a single consumer to prevent duplicate processing.

Advantages over message-based load balancing

When to use

Queue-based load balancing is best for stream computing and data aggregation applications that need to batch-process or aggregate messages from the same source. For example, a consumer might collect metrics from a single queue over a time window and compute running averages.

Scope

Queue-based load balancing is the only policy available for consumers on broker versions 3.x and 4.x, including Pull Consumer, default Push Consumer, default Pull Consumer, and Lite Pull Consumer types. On broker version 5.x, Pull Consumer types continue to use queue-based load balancing by default.

No additional configuration is required -- queue-based load balancing is enabled automatically for these consumer types.

Example

For sample code, see LitePullConsumerAssign.java in the Apache RocketMQ code library.

Implement idempotent message handling

Both load balancing policies trigger temporary rebalancing when consumers are added, removed, or when the broker scales. During rebalancing, a small number of messages may be delivered more than once. To handle this, implement deduplication to ensure idempotence for your message consumption logic.