Symptoms

A consumer offset of a consumer group expires, but the consumer offset is not deleted. The consumer offset is still displayed on the Subscriptions tab in the Message Queue for Apache Kafka console.

Note The Consumer Offset Retention Period parameter specifies the maximum period that consumer offsets can be retained. The retention period of a consumer offset starts from the point in time when the consumer offset is committed. After the retention period expires, the consumer offset is deleted. Each time a consumer offset is committed to a topic partition, the retention period is reset for recalculation. For more information, see Modify the message configurations.

Causes

The consumer group is active. In this case, the consumer offsets of the consumer group are not deleted, regardless of whether the retention period of the consumer offsets exceeds the maximum retention period. This is a mechanism designed by open source Kafka.

Note A consumer group that contains an active consumer thread is considered to be active.

For example, you create a consumer group named TestGroup to subscribe to Topic A, and consumer offsets are committed. Due to business adjustment, you cancel the subscription to Topic A and subscribe to Topic B. In this case, the TestGroup consumer group is active, and the consumer offsets for Topic A and Topic B are not deleted.

If you cancel the subscription to Topic A and do not subscribe to Topic B, the TestGroup consumer group is not active. In this case, the consumer offsets whose retention period exceeds the maximum retention period are deleted.

If you do not want to wait until a consumer offset expires and is deleted, you can use a new consumer group instead of the TestGroup consumer group to subscribe to Topic B. If you can wait until a consumer offset expires and is deleted, you can use the TestGroup consumer group to subscribe to Topic B after the consumer offsets for Topic A are deleted.

Solutions

Determine whether to create another consumer group to subscribe to a new topic based on your business needs.
  • If you do not want to wait until a consumer offset expires and is deleted, use a new consumer group instead of the existing one to subscribe to a new topic.
  • If you can wait until a consumer offset expires and is deleted, you can use the existing consumer group to subscribe to a new topic after the existing consumer offsets are deleted.