When a consumer rebalance occurs, you can view its details, including the time, duration, count, cause, and whether new members have joined the consumer group.
Background
A rebalance is the process where ApsaraMQ for Kafka redistributes partition assignments among the consumers in a consumer group. A rebalance can be triggered for the following reasons:
A consumer subscribing to or unsubscribing from a topic triggers a rebalance within the consumer group.
A change in the number of partitions for a topic triggers a rebalance.
The number of consumers in a consumer group increases or decreases. This can happen when:
A consumer's heartbeat times out due to stalled message processing, triggering a rebalance to remove the unhealthy consumer.
If message processing is too slow, a consumer may fail to call
poll()within the time specified bymax.poll.interval.ms(default is 5 minutes). Consequently, the client leaves the group, triggering a rebalance.Shutting down surplus consumers to conserve resources triggers a rebalance.
Adding more consumers to handle processing delays caused by an under-provisioned group triggers a rebalance.
Procedure
Log on to the ApsaraMQ for Kafka console. In the left-side navigation pane, click Instances.
In the top navigation bar, select the region where the instance that you want to manage resides. On the Instances page, click the name of the instance that you want to manage.
In the left-side navigation pane, click Groups, and then click the name of the target group.
On the Group Details page, click the Rebalance Details tab.
If New Members Joined shows Yes, you can click View Details.
Member information
Parameter | Description |
Member type | Existing member: A consumer that maintains an active connection. New member: A newly joined consumer. Leaving member: A consumer that has disconnected. Members are grouped to help you identify whether the rebalance was caused by consumers joining or leaving. |
Consumer join time | The time when the consumer rejoined the group. |
Blocking time (ms) | Rebalance start time minus consumer join time. |
SessionTimeout (ms) | The timeout duration that the broker uses to determine whether a consumer is alive. |
RebalanceTimeout (ms) | The maximum time the broker waits for all consumers to rejoin the group during a rebalance. |
MemberId | The unique identifier assigned by the broker to each consumer instance. It is generated and returned by the broker when the consumer sends its first JoinGroup request. |
ClientId | The client.id configured on the client side. |
ClientIp | The IP address of the client. |
ApiVersion | The version of the Kafka protocol API used for communication between the Kafka client and the broker. |
ProtocolType | The partition assignment protocol type used by the consumer group. Valid values: |
GroupInstanceId | The static member identifier of the consumer, which is manually specified by the user. When a consumer restarts with the same GroupInstanceId, it can skip the rebalance process. |
Troubleshoot frequent rebalances
Possible causes
Frequent rebalances are often caused by one of the following issues, depending on your client version:
Clients earlier than version 0.10.2: The consumer does not have a separate heartbeat thread. Heartbeats are sent through the
poll()method. If message processing takes too long, the heartbeat request times out and a rebalance is triggered.Clients of version 0.10.2 or later: A separate heartbeat thread exists. However, if no messages are pulled after the time specified by
max.poll.interval.mselapses, the client leaves the consumer group and a rebalance is triggered. The default value ofmax.poll.interval.msis 5 minutes.
Solutions
Understand the following parameters and concepts:
session.timeout.ms: The heartbeat timeout, configured on the client.max.poll.records: The maximum number of messages returned per poll.Clients earlier than v0.10.2: Heartbeats are sent via the
poll()API, as these clients lack a dedicated heartbeat thread.Clients v0.10.2 and later: These versions use the
max.poll.interval.msparameter to prevent a consumer from pausing message processing for an extended period.
Tune parameter values
Configure the following parameters based on your client version:
session.timeout.msClient version Recommended value Earlier than 0.10.2 Larger than the time to process a batch of messages, but no greater than 30 seconds. 25 seconds is recommended. 0.10.2 or later Keep the default value of 10 seconds. max.poll.recordsSet this value to be far smaller than the result of the following formula:
max.poll.records << messages_per_thread_per_second * number_of_threads * max.poll.interval.msmax.poll.interval.ms(version 0.10.2 or later only)Set this value to be larger than the result of the following formula:
max.poll.interval.ms > max.poll.records / (messages_per_thread_per_second * number_of_threads)Improve consumption speed and separate processing threads
Improve your message processing speed by allocating a separate thread for consumption logic. This prevents slow processing from blocking heartbeats or exceeding the poll interval.
Reduce topics per consumer group
Reduce the number of topics that each consumer group subscribes to. Subscribe to no more than five topics per consumer group. For optimal stability, subscribe to one topic per consumer group.
Upgrade to version 0.10.2 or later
If you are using a client version earlier than 0.10.2, upgrade to version 0.10.2 or later. Later versions use a separate heartbeat thread, which prevents processing delays from causing heartbeat timeouts.
Why can't I see member information for my instance?
Version requirements:
Subscription and pay-as-you-go instances: Version 5.2.2.8 or later is required.
Serverless instances: All versions are supported.