Problem description
You are using an ApsaraMQ for Kafka instance with a server-side version of 2.2.0. The consumer offsets of a Group are retained even after they have expired. You can still view the consumer details for the topic on the Subscriptions tab of the Topic Details page in the ApsaraMQ for Kafka console by clicking Consumer Details.
The Consumer Offset Retention Period parameter specifies the maximum duration for which consumer offsets can be retained. The retention period for a consumer offset starts when the offset is committed. When the retention period expires, the system deletes the consumer offset. Note that deleting a consumer offset does not reset the offset to 0. Instead, it removes the offset record. Each time a consumer offset is committed to a topic partition, the retention period timer for that consumer offset is reset to 0. For more information, see Modify message configurations.
Possible cause
According to the design of open source Kafka, the consumer offsets of an active Group are not deleted even if they exceed the specified retention period.
A Group is active if it contains a consumer thread that uses the `subscribe` method, which is different from the `assign` method.
For example, you create a Group named TestGroup to subscribe to Topic A and successfully commit its consumer offsets. Due to business changes, you unsubscribe TestGroup from Topic A and subscribe it to a new topic, Topic B. In this case, TestGroup remains active. The consumer offsets for the Group on both Topic A and Topic B are not deleted.
If you only unsubscribe TestGroup from Topic A and do not subscribe it to a new topic, TestGroup becomes inactive. The consumer offsets for Topic A are then deleted after their retention period expires.
Solutions
To delete expired consumer offsets, perform one of the following operations:
Disconnect the Group to make it inactive. Then, perform one of the following operations:
Wait for the consumer offsets to expire. The system will then automatically delete them.
Reset the expired consumer offsets to 0 for each partition in the console.
Migrate the consumers to a new Group. This makes the original Group inactive. Wait for the consumer offsets of the original Group to expire. The system will then automatically delete them.
Upgrade the server-side version to 2.6.0 or later.
With this version, you do not need to disconnect consumers. You can simply unsubscribe from topics that are no longer needed. For example, if TestGroup unsubscribes from Topic A but remains active with a subscription to Topic B, the consumer offset for the combination of TestGroup and Topic A is cleared after its retention period expires.