Sarama Go client issues ApsaraMQ Kafka - ApsaraMQ for Kafka

Problem description

The Sarama Go client has the following known compatibility issues with ApsaraMQ for Kafka that can affect message production and consumption:

New partitions are not automatically detected. When you add partitions to a topic, the client may fail to detect or consume from them until restarted.
Non-standard protocol behavior. The Sarama Go client does not fully comply with Apache Kafka protocol standards. During broker exceptions, this can cause:
- Unexpected offset reset (OutOfRange mechanism). If auto.offset.reset is set to Oldest(earliest), the client may reconsume all messages from the earliest offset, causing large-scale duplicate processing.
- Persistent Rebalance state. The client may get stuck in the Rebalance state and stop consuming messages.

Upgrade the Sarama Go client to version 1.41.0 or later, or replace it with a Confluent Go client.

A Confluent Go client demo is available at kafka-confluent-go-demo.

Important

If you cannot replace the client immediately, reduce risk with these measures:

Set the consumer offset reset policy appropriately. In production, set auto.offset.reset to Newest(latest). Use Oldest(earliest) only in debugging environments or when large-scale duplicate messages are acceptable.
Reset consumer offsets from the console if a backlog occurs. If an offset reset causes a significant message backlog, reset the consumer offset to a specific point in time in the ApsaraMQ for Kafka console. No code changes or consumer group modifications are required. Reset consumer offsets.