ApsaraMQ for Kafka:Best practices for consumers

How consumer groups work

ApsaraMQ for Kafka distributes messages across consumers through consumer groups. Each consumer in a group is assigned a subset of partitions from the subscribed topics, and each partition is consumed by exactly one consumer in the group.

The consumption cycle follows three steps:

1. Poll messages from assigned partitions.

2. Process the messages (run your consumption logic).

3. Poll again.

To place multiple consumers into the same group, set their group.id to the same value. ApsaraMQ for Kafka then evenly distributes partitions across those consumers so that each message is delivered to exactly one consumer in the group.

For example, if Group A subscribes to Topic A and consumers C1, C2, and C3 are started in the group, each message that Topic A receives is delivered to only one of C1, C2, or C3. By default, ApsaraMQ for Kafka evenly delivers messages to consumers to balance the consumption load.

A rebalance -- the process of redistributing partitions across consumers -- is triggered whenever consumers are first started, restarted, added, or removed.

Set the partition count

The partition count determines the maximum number of concurrent consumers for a topic.

By default, ApsaraMQ for Kafka sets the partition count to 12 in the console, which is sufficient for most workloads. If you need higher throughput, increase the count gradually within the recommended range of 12 to 100:

Prevent frequent rebalances

Rebalances pause consumption across the entire consumer group while partitions are reassigned. Frequent rebalances are typically caused by heartbeat timeouts: the broker considers a consumer dead when it stops receiving heartbeats within the session timeout window.

To prevent rebalances, you can adjust the related parameters or increase the consumption rate. For detailed troubleshooting, see Why do rebalances frequently occur on my consumer client?

Choose a subscription mode

ApsaraMQ for Kafka supports two subscription patterns.

String topicStr = kafkaProperties.getProperty("topic");
String[] topics = topicStr.split(",");
for (String topic: topics) {
    subscribedTopics.add(topic.trim());
}
consumer.subscribe(subscribedTopics);

Use one group per application

Use a separate consumer group for each application. Each application typically has distinct consumption logic, so sharing a group across applications leads to unpredictable behavior.

If you need different consumption logic within the same application, create separate kafka.properties files (for example, kafka1.properties and kafka2.properties) and initialize each consumer with its own configuration.

Keep subscriptions consistent

Within a consumer group, we recommend that all consumers subscribe to the same set of topics. Inconsistent subscriptions across consumers in the same group can cause unexpected partition assignment and make troubleshooting difficult.

Consumer offsets

Every partition tracks a maximum offset -- the total number of messages produced to that partition. Each consumer tracks a consumer offset -- the position of the last consumed message. The difference between these two values represents the message backlog (accumulated unconsumed messages).

Choose an offset commit strategy

ApsaraMQ for Kafka supports three approaches for committing consumer offsets. The strategy you choose directly affects delivery semantics and data correctness.

// Set enable.auto.commit=false in your consumer properties

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        // Process the record
    }
    consumer.commitSync(); // Commit only after successful processing
}

// Set enable.auto.commit=false in your consumer properties

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        // Process the record
    }
    consumer.commitAsync((offsets, exception) -> {
        if (exception != null) {
            log.error("Commit failed for offsets: {}", offsets, exception);
        }
    });
}

Configure offset reset behavior

Consumer offsets are reset in two situations:

• No offset has been committed to the broker -- for example, when a consumer group connects for the first time.

• The consumer tries to fetch from an invalid offset -- for example, the stored offset exceeds the current maximum.

Control reset behavior with auto.offset.reset on Java clients:

Handle large messages

Consumers pull messages from the broker. For messages larger than 1 MB, tune these parameters to control the pull rate:

With these settings, the consumer pulls large messages one at a time, preventing memory issues.

Deduplicate messages

ApsaraMQ for Kafka uses at-least-once delivery semantics: every message is delivered at least once, guaranteeing no data loss. However, network errors or client restarts can cause a small number of messages to be delivered more than once.

For applications sensitive to duplicates (such as transaction processing), implement idempotent consumption:

1. On the producer side: Set a unique key for each message that serves as a transaction ID.

2. On the consumer side: Before processing a message, check whether its key has already been processed. Skip the message if it has; otherwise, process it exactly once.

If your application tolerates occasional duplicates, no deduplication logic is needed.

Handle consumption failures

Messages within a partition are consumed sequentially. If processing fails for a specific message (for example, due to corrupt data), use one of these strategies:

Topic A (source) --> Consumer --> Processing
                                    |
                              (on failure)
                                    |
                                    v
                        Dead-letter topic --> Manual review

ApsaraMQ for Kafka does not provide built-in dead-letter queue semantics. Implement this pattern by producing failed messages to a separate topic and reviewing them periodically.

Prevent message accumulation

Message accumulation is the most common consumer-side issue. Two root causes:

Increase consumption rate

Filter messages

ApsaraMQ for Kafka does not provide built-in message filtering. Two alternatives:

Combine both approaches as needed.

Simulate message broadcasting

ApsaraMQ for Kafka does not support native message broadcasting. To deliver every message to multiple independent consumers, create a separate consumer group for each consumer. Each group independently receives all messages from the subscribed topic.

Reduce consumption latency

Latency stays low as long as consumers keep up with the production rate. If latency increases, check for message accumulation and increase the consumption rate.