ApsaraMQ for Kafka:Why do alert and console accumulations differ?

Cause 1

Message accumulation for a partition is calculated as: partition maximum offset - partition consumer offset. The total message accumulation is the sum of the message accumulation across all partitions.

For example, consider an instance with m consumer groups and n topics, where each consumer group subscribes to all topics. Message accumulation for a consumer group can be calculated in two ways:

• Standard request method

  This method sends an RPC request to get the consumer offsets for the consumer group, and another RPC request to get the maximum offsets for the partitions of the subscribed topics. This method generates at least m * n * number of broker nodes RPC requests, which significantly impacts monitoring efficiency.

• Batch request method

  This method first fetches all consumer offsets for all consumer groups in a batch. Then, another batch request fetches the maximum offsets for all subscribed partitions. This approach reduces the number of RPC requests from m * n * number of broker nodes to just the number of broker nodes. However, due to the nature of batch processing, a delay occurs between fetching the consumer offsets and the maximum offsets. During this delay, the maximum offset of a partition can continue to increase as new messages are produced. This time gap leads to an inaccurate calculation of message accumulation.

The Total Message Accumulation on the Group Details page in the console is obtained by making separate RPC requests to retrieve the consumer offset and the maximum partition offset. Because the time difference between these two RPC requests is small, the discrepancy from the actual message accumulation is also small. The message accumulation for monitoring is retrieved by using batch requests. This method creates a time lag between when the consumer offset is retrieved and when the maximum partition offset is retrieved, which inevitably causes a discrepancy in the calculated accumulation. This is why the value for the message accumulation alert may be inconsistent with the console message accumulation.

Cause 2

When consumption is slow and disk usage is high, the instance may delete messages before they are consumed. This can cause a partition's consumer offset to become smaller than its minimum offset. open source Apache Kafka and Message Queue for Apache Kafka handle this situation differently:

• open source Apache Kafka

  By default, it ignores this message accumulation when it calculates totals.

• Message Queue for Apache Kafka

  To help you detect abnormal consumer offsets in partitions faster, the monitoring alert policy reports them as message accumulation alerts, allowing you to handle them as needed.

To help you distinguish between abnormal and normal partitions in the console, Message Queue for Apache Kafka excludes the message accumulation of abnormal partitions from the Total Message Accumulation on the Group Details page. Therefore, the message accumulation reported by the monitoring system is much larger than the console message accumulation. This causes a discrepancy between the message accumulation alert and the console message accumulation.

The following examples illustrate this behavior:

• The console's total message accumulation and the alert's total message accumulation do not match.

  On the Kafka consumer group details page, select the Prometheus monitoring tab. The Total Message Accumulation at the top shows 13,568, while the Total Message Accumulation line chart below shows that the accumulation has remained stable over the last 30 minutes.

• The console's total message accumulation excludes abnormal accumulation.

  On the consumer group details page, the Total Message Accumulation is displayed in a card at the top. On the Consumption Status tab, a table shows the accumulation for each topic in the Accumulation column.

• The consumption details for a topic show partitions with abnormal accumulation where the consumer offset is less than the minimum offset.

  For example, partition 0 has a Consumer Offset of 1, but its Min Offset is 97,413,417,052. The Accumulation is displayed as -- and is not included in the total.