ApsaraMQ for Kafka deletes consumer offsets after they exceed the configured consumer offset retention period. However, the consumer offsets of an active Group are retained even after they expire. This topic explains the cause and provides solutions.
Symptom
On a 2.2.0 ApsaraMQ for Kafka server, the consumer offsets of a Group exceed the consumer offset retention period that you configured, but they are not deleted. You can still view these consumer offsets in the console: open the Topic Details page in the ApsaraMQ for Kafka console, click the Subscriptions tab, and view the Consumer Details of the target topic.
Consumer Offset Retention Period specifies the maximum period for which a consumer offset is retained. The period is counted from the time the consumer offset is committed. Consumer offsets that exceed the period are deleted. Deleting a consumer offset does not set the offset to 0. Instead, no consumer offset exists. Each time you commit a consumer offset to a topic partition, the retention timer of that consumer offset resets to 0. For more information, see Modify message configurations.
Possible causes
Based on the open source Kafka design, the consumer offsets of an active Group are not deleted, even if the consumer offsets exceed the consumer offset retention period.
A Group is an active Group if it contains a consumer thread that uses the subscribe method, as opposed to the assign method.
For example, you create a Group named TestGroup to subscribe to TopicA, and you commit consumer offsets. Because of a business change, you unsubscribe from TopicA and subscribe to TopicB. TestGroup stays active, so the consumer offsets of this Group on both TopicA and TopicB are not deleted.
If you unsubscribe only from TopicA and do not subscribe to TopicB, TestGroup becomes inactive. The consumer offsets of TestGroup on TopicA are deleted after they exceed the consumer offset retention period.
Solutions
To delete expired consumer offsets, use one of the following methods:
Disconnect all consumer threads that use the
subscribemethod to make the Group inactive, and then use one of the following methods based on your business requirements:Wait for the consumer offsets to expire. After they expire, the consumer offsets are deleted automatically.
Reset the expired consumer offsets to 0 by partition in the console.
Migrate to a new Group, make all consumer threads of the original Group inactive, and wait for the consumer offsets of the original Group to expire. After they expire, the consumer offsets are deleted automatically.
Upgrade the server version to 2.6.0.
Keep the connection of the Group and unsubscribe from the topics that your business no longer needs. Even if the Group remains active, the consumer offsets of the unsubscribed topics are cleared after they exceed the consumer offset retention period.
After you apply one of the preceding methods, verify the result in the console. Open the Topic Details page, click the Subscriptions tab, and view the Consumer Details of the topic. If the consumer offsets are deleted, no consumer offset exists. If you reset the consumer offsets to 0, the consumer offsets remain with value 0.