All Products
Search
Document Center

The consumer of Message Queue for Apache Kafka cannot pull messages from the server or the messages are pulled slowly

Last Updated: Sep 25, 2020

Problem description

If existing messages exist in the Topic and the messages are not consumed by the Consumer, the following exceptions may occur (especially when the messages are consumed over the Internet):

  • The consumer cannot pull the message from the server.
  • Consumer pulls messages from the broker slowly.

Causes

The possible causes of the problem are as follows:

  • The consumption traffic of the instance exceeds the peak bandwidth.
  • The size of a single message in the instance Topic exceeds the peak bandwidth.
  • The number of messages pulled by the Consumer exceeds the peak bandwidth.

Solution

Follow these steps to troubleshoot the issue. If your instance is a public network instance, the peak bandwidth in the article is replaced by public network traffic.

  1. Log on to the Message Queue for Apache Kafka console. Click monitoring and alerting to view the monitoring information. For more information about how to view or set monitoring alarm information, see monitoring and alerting.
  2. Check consumption traffic of instance messages indicates whether the peak bandwidth of the instance has been reached. If the peak bandwidth is reached, you must upgrade the peak bandwidth. For more information about how to upgrade the instance configuration, see upgrade instance specifications.
  3. Check whether the size of a single message in the Topic exceeds the peak bandwidth. If so, you can upgrade the bandwidth peak of the instance or reduce the size of individual messages. For information on how to reduce the size of a single message, see change message configuration.
  4. Check whether the number of messages pulled by the Consumer each time exceeds the peak bandwidth. If the latency exceeds the configured value, messages cannot be pulled. You need to adjust the following parameters to reduce the number of messages pulled for each consumption.
    • max.poll.records: the maximum number of messages pulled at a time.
    • fetch.max.bytes: the maximum number of bytes in each pull, which must be less than the peak bandwidth.
    • max.partition.fetch.bytes: the maximum number of bytes in each Partition pulled. Ensure that ( max.partition.fetch.bytes) * (total number of subscription partitions) <the peak bandwidth.

Application scope

  • Message Queue for Apache Kafka