Troubleshoot frequent consumer rebalances in ApsaraMQ for Kafka - ApsaraMQ for Kafka

Symptom

Rebalances occur frequently on your consumer client when you use ApsaraMQ for Kafka.

Cause

The root cause depends on your client version, because the heartbeat mechanism differs between versions.

Clients earlier than version 0.10.2: There is no dedicated heartbeat thread. Heartbeats are sent inside the poll() call. When message processing takes longer than session.timeout.ms, the broker receives no heartbeat, removes the consumer from the group, and triggers a rebalance.
Clients of version 0.10.2 or later: A dedicated heartbeat thread runs independently of message processing. However, if the consumer does not call poll() within the interval set by max.poll.interval.ms (default: 5 minutes), the client leaves the consumer group and a rebalance is triggered. The default value of max.poll.interval.ms is 5 minutes. This typically happens when processing a batch of messages takes too long.

The following error messages are commonly associated with rebalances. Identifying the specific error can help you locate the root cause more quickly:

NOT_COORDINATOR / join group failed: Indicates that the Group Coordinator has changed (for example, due to broker failover) or that a heartbeat timeout caused the consumer to be removed from the group. The consumer attempts to rejoin the group, triggering a rebalance.
MemberIdRequiredException / consumer poll timeout: For clients earlier than version 0.10.2, this is typically caused by a stalled consumer that blocks the heartbeat. For clients of version 0.10.2 or later, this indicates that slow message processing caused the consumer to exceed max.poll.interval.ms without calling poll(), prompting the client to leave the consumer group.
CommitFailedException: Indicates that the consumer took too long to process messages or failed to send heartbeats in time. The broker removes the consumer from the group (triggering a rebalance) before the offset commit can complete.

Key parameters

The following parameters control rebalance behavior:

Parameter	Applicable versions	Description
`session.timeout.ms`	All	Session timeout. If no heartbeat is received within this period, the broker removes the consumer from the consumer group.
`max.poll.records`	All	The maximum number of messages returned per call to the `poll()` method.
`max.poll.interval.ms`	0.10.2 or later	The maximum interval between two consecutive calls to the `poll()` method. If this interval is exceeded, the consumer leaves the consumer group and a rebalance is triggered.
`heartbeat.interval.ms`	0.10.2 or later	The interval at which the consumer sends heartbeats to the broker. Must be less than `session.timeout.ms`. A value of 10,000 milliseconds (10 seconds) or less is recommended.

Solutions

Tune parameter values

Configure the following parameters based on your client version:

session.timeout.ms

Client version	Recommended value
Earlier than 0.10.2	Larger than the time to process a batch of messages, but no greater than 30 seconds. 25 seconds is recommended.
0.10.2 or later	Keep the default value of 10 seconds.

max.poll.records

Set this value well below the result of the following formula:

max.poll.records << messages_per_thread_per_second * number_of_threads * max.poll.interval.ms

max.poll.interval.ms (version 0.10.2 or later only)

Set this value above the result of the following formula:

max.poll.interval.ms > max.poll.records / (messages_per_thread_per_second * number_of_threads)

heartbeat.interval.ms (version 0.10.2 or later)

Set heartbeat.interval.ms to no more than 10,000 milliseconds (10 seconds). This value controls how frequently the consumer sends heartbeats to the broker and must always be less than session.timeout.ms. A common practice is to set it to one-third of session.timeout.ms.

Configuring parameters in Spring Kafka

If you use Spring Kafka, you can configure all of the above parameters in your application.yml file:

spring:
  kafka:
    consumer:
      properties:
        session.timeout.ms: 10000
        heartbeat.interval.ms: 3000
        max.poll.interval.ms: 300000
        max.poll.records: 50

Adjust the values to match your consumption throughput. The max.poll.interval.ms value must exceed the time required to process a single batch of max.poll.records messages.

Improve consumption speed and separate processing threads

Offload message processing to a dedicated thread so the consumer thread can call poll() on schedule. This prevents slow processing from blocking heartbeats or exceeding the poll interval.
Reduce topics per consumer group

Keep each consumer group subscribed to no more than five topics. For optimal stability, subscribe to one topic per consumer group.
Upgrade to version 0.10.2 or later

Clients earlier than version 0.10.2 lack a dedicated heartbeat thread, making them vulnerable to heartbeat timeouts during heavy processing. Upgrading to version 0.10.2 or later decouples heartbeating from message processing and eliminates this class of rebalance.

FAQ

Q: How do I temporarily reduce rebalances to speed up consuming backlogged messages?

Take the following steps to minimize rebalances and clear a message backlog as quickly as possible:

Use client version 0.10.2 or later. Clients earlier than version 0.10.2 lack a dedicated heartbeat thread and are prone to rebalances under heavy load. Upgrade before tuning other parameters.
Increase max.poll.interval.ms to a value greater than the maximum time required to process a single batch of messages. This prevents the consumer from being considered unresponsive during heavy processing.
Decrease max.poll.records to reduce the number of messages fetched per poll() call. Smaller batches finish faster, reducing the risk of exceeding max.poll.interval.ms.
Reduce the number of topics subscribed per consumer group. Keep subscriptions to five topics or fewer. For optimal stability, subscribe to one topic per consumer group.
Avoid blocking the consumer thread. Move message processing logic to a separate asynchronous thread so the consumer thread remains free to call poll() on schedule.