Kafka Consumer Lag Overview: Causes and Impact - ApsaraMQ for Kafka

Message accumulation in ApsaraMQ for Kafka occurs when consumers cannot keep up with producers, causing unprocessed messages to pile up in topic partitions. This metric is commonly called consumer lag.

Left unaddressed, message accumulation increases processing latency, triggers rebalancing loops, and can cause out-of-memory (OOM) errors on consumer clients.

How message accumulation works

Each partition in a topic tracks two offsets:

Consumer offset -- the position up to which a consumer group has processed messages.
Latest offset -- the position of the most recently produced message.

The gap between these two values is the accumulation for that partition:

Accumulation per partition = Latest offset - Consumer offset
Total accumulation = Sum of accumulation across all partitions

An accumulation near zero means consumers are keeping pace with producers. A growing accumulation signals that consumers are falling behind.

Example

Topic: test (Partition 0)
+----+----+----+----+----+----+----+
| M1 | M2 | M3 | M4 | M5 | M6 | M7 |   <- 7 messages written
+----+----+----+----+----+----+----+
          ^                    ^
    Consumer offset (3)   Latest offset (7)

Topic: test (Partition 1)
+----+----+----+----+----+----+
| M1 | M2 | M3 | M4 | M5 | M6 |   <- 6 messages written
+----+----+----+----+----+----+
          ^                ^
    Consumer offset (3)   Latest offset (6)

Total accumulation = (7 - 3) + (6 - 3) = 7 messages

Note

If a consumer offset does not exist -- because the consumer has not committed an offset or the offset has expired -- and at least one consumer thread in the Group is online, accumulation is calculated as Latest offset - Earliest offset across all partitions. If all consumer threads in the Group are offline, accumulation is 0.

Diagnose the root cause

Start by determining whether consumers are running:

Consumers are offline or restarting

Symptom	Likely cause
Consumer processes are not running	Crashes, deployment updates, or long garbage collection (GC) pauses
Consumers cycle between online and offline	Frequent rebalancing caused by consumers joining or leaving the Group, heartbeat timeouts, or a low `session.timeout.ms` value

Consumers are running but falling behind

Symptom	Likely cause
Steady increase in accumulation	Insufficient consumer throughput -- slow I/O, CPU or memory bottlenecks, or per-message processing logic is too slow
Sudden spike in accumulation	Traffic surge from peak load or batch imports
Accumulation despite adequate resources	Code issues such as infinite loops, uncaught exceptions, or long intervals between `poll()` calls
Accumulation at a constant rate	Consumer rate limiting -- the consumption rate has reached the reserved or elastic limit of the instance
Accumulation that appears larger than expected	Delayed or failed offset commits, causing repeated pulls and inflated lag numbers

Monitor accumulation metrics

View accumulation metrics based on your instance type:

Instance type	Monitoring tool
Subscription or pay-as-you-go	Prometheus monitoring
Serverless	Dashboard

Impact of unresolved accumulation

Unresolved accumulation affects the broader system in several ways:

Increased latency -- Delayed processing affects downstream services and business decisions.
Blocked threads and timeouts -- Overwhelmed consumers may block, causing request timeouts and tripping circuit breakers.
Rebalancing loops -- Slow processing leads to heartbeat timeouts, which trigger partition rebalancing. Rebalancing pauses consumption, increases repeated pulls, and worsens lag -- creating a negative feedback loop.
OOM errors -- If a consumer calls poll() but does not process messages fast enough, unprocessed messages accumulate in the client's memory buffer and can cause a heap overflow.

Resolve message accumulation

Scale consumer throughput

Add consumer instances -- Add more consumers to the same Group. Each partition is assigned to at most one consumer within a Group, so the number of consumers must not exceed the number of partitions (partitions >= consumers).
Increase partitions -- More partitions allow higher parallelism. Each new partition can be assigned to a separate consumer.
Process messages asynchronously -- Offload time-consuming operations (database writes, API calls) to a thread pool or task queue so that poll() returns quickly.
Use batch processing -- Process multiple messages per loop iteration instead of one at a time.

Tune consumer parameters

Parameter	Default	Recommended range	Purpose
`max.poll.records`	500	1 -- 500	Controls how many records each `poll()` call returns. Lower values reduce per-poll processing time.
`fetch.min.bytes`	1 B	1 KB -- 1 MB	Sets the minimum data the broker collects before responding to a fetch request. Higher values reduce empty fetches and improve throughput.
`fetch.max.wait.ms`	500 ms	500 ms	Sets the maximum time the broker waits to accumulate `fetch.min.bytes` before responding.
`session.timeout.ms`	10 s	30 s	Maximum time the broker waits without a heartbeat before declaring a consumer dead. A higher value prevents false evictions.
`heartbeat.interval.ms`	3 s	<= `session.timeout.ms` / 3	Controls heartbeat frequency. Must be lower than `session.timeout.ms` to maintain group membership.
`enable.auto.commit`	false	true	Enables automatic offset commits. Prevents offset commit failures from causing false accumulation.

Emergency measures

If accumulation is too large to drain through normal processing, reset the consumer offset to the latest position. This skips all accumulated messages and resumes consumption from the most recent offset.

Warning

Resetting the consumer offset to the latest position discards all accumulated messages. Only use this approach when the accumulated messages are no longer needed.

To clear accumulation-based alerts, use ApsaraMQ for Kafka to reset the consumer offset of a topic partition to 0. When the consumer offset is 0, the reported accumulation is 0.