All Products
Search
Document Center

The consumption time of partitions in Message Queue for Apache Kafka is different

Last Updated: Sep 25, 2020

Problem description

The consumption time of partitions in Message Queue for Apache Kafka is different, as follows:

  • The consumption time of each partition varies greatly.
  • The consumption time in some partitions lags behind significantly.
  • The consumption time is out of order in some shards.

Causes

Currently, on the Consumer groups page of the Message Queue for Apache Kafka console, the consumption time you check is the last consumed message storage time in this partition, instead of the consumption time of this message. The latest consumption time for each partition is different. This may be because the partitions receive the Producer message at different times. For example, if a Shard always receives messages from the producer first, the latest consumption time of the Shard is earlier than that of other shards.

Note: For more information about how to view the message accumulation status, see consumption status.

Why do partitions receive producer messages at different times? The reasons are as follows:

All Consumer instances (N) in the same Consumer Group are distributed based on the number of partitions (M). The two message types are described as follows:

  • Number of partitions M can be divided consumer number of instances N: specifies that the number of consumer allocated to messages is uniform. For example, if the number of partitions M is 24 and the number of consumer instances N is 6, each consumer instance will consume four partitions (24 = 6*4). Whether messages are evenly consumed depends on whether messages sent by the producer are evenly distributed to each partition.
  • Number of partitions M cannot be exactly divided consumer number of instances N: then the consumer allocated to the message is uneven. For example, if the number of shards M is 24, consumer the number of instances is 5, then 4 consumer instances consume 5 partitions each, and the remaining 1 consumer instance consumes 4 partitions, that is, 24 =( 4*5 ) + ( 1*4). If the processing performance of each consumer is consistent, the consumption speed of a consumer instance with five partitions is slower than that of a consumer instance with four partitions.

Solution

According cause of problem the consumption time inconsistency of each partition has a variety of factors, such as the fractional partition, the number of consumer instances, the number of messages sent by the producer, and the processing performance of each consumer. Therefore, it is not an error condition and can be ignored.

Application scope

  • Message Queue for Apache Kafka