ApsaraMQ for Kafka:Client parameter configuration

acks

Controls how many broker acknowledgments the producer requires before considering a send successful.

Recommended value: 1 for most workloads that prioritize throughput over strict durability.

retries

Maximum number of times the producer retries a failed send. A higher value helps the producer recover from transient broker failures such as leader elections. Combine with retry.backoff.ms to control retry pacing.

retry.backoff.ms

Delay between send retries. A value that is too low can cause retry storms during broker failovers.

batch.size

Maximum size of a single batch per partition. When a batch reaches this size, the producer sends it immediately.

Keep the default value of 16384 for most workloads. A smaller value increases the number of network requests and reduces throughput. If you increase batch.size, make sure buffer.memory is large enough to accommodate the larger batches.

linger.ms

Maximum time the producer waits for a batch to fill before sending it. This works like Nagle's algorithm in TCP: once a batch reaches batch.size, it is sent immediately regardless of the linger timer. If the batch is still below batch.size when linger.ms elapses, the producer sends whatever has accumulated.

A higher linger.ms increases batching efficiency and throughput at the cost of per-message latency.

buffer.memory

Total memory the producer allocates for buffering unsent records across all partitions. If this pool is exhausted, send() blocks or throws an exception depending on max.block.ms. An undersized buffer causes slow memory allocation, reduced throughput, or send timeouts.

Unit: bytes. Default: 33554432 (32 MB).

Sizing formula:

buffer.memory >= batch.size x number_of_partitions x 2

For example, with batch.size=16384 and 50 partitions:

16384 x 50 x 2 = 1,638,400 bytes (~1.6 MB minimum)

If you increase batch.size for throughput, scale buffer.memory proportionally.

partitioner.class

Determines how the producer assigns records to partitions. The sticky partitioning strategy reduces the number of incomplete batches by filling one partition's batch before moving to the next.

If your producer client is earlier than version 2.4, explicitly set the sticky partitioner to improve batching efficiency.

fetch.min.bytes

Minimum amount of data the broker accumulates before returning a fetch response. A larger value reduces fetch frequency and broker CPU overhead, which improves throughput but increases end-to-end message latency. Unit: bytes.

Evaluate your producer's message rate before setting this value. If messages arrive slowly, a large fetch.min.bytes adds unnecessary delay.

fetch.max.wait.ms

Maximum time the broker waits to accumulate fetch.min.bytes before returning a response. Unit: milliseconds.

Behavior varies by storage type:

• Local storage: The broker waits until fetch.min.bytes is reached or fetch.max.wait.ms elapses, whichever comes first.

• Cloud storage: The broker returns a response immediately when new data arrives, regardless of fetch.min.bytes.

max.partition.fetch.bytes

Maximum amount of data the broker returns per partition in a single fetch. Unit: bytes.

session.timeout.ms

Maximum time between heartbeats before the broker considers the consumer dead and triggers a rebalance.

max.poll.records

Maximum number of records returned in a single poll() call. If the consumer cannot process this many records before the next poll() deadline, the broker considers it dead and triggers a rebalance.

Sizing formula:

max.poll.records < messages_per_thread_per_second x consumer_threads x session_timeout_seconds

For example, with 500 msg/s per thread, 4 threads, and a 45-second session timeout:

500 x 4 x 45 = 90,000

Set max.poll.records below this value to ensure the consumer always finishes processing before the session times out.

max.poll.interval.ms

Maximum interval between consecutive poll() calls before the broker removes the consumer from the group. This parameter applies only to Java client 0.10.1 and later, where heartbeats run on a separate thread.

Sizing formula:

max.poll.interval.ms > time_per_record x max.poll.records

In most cases, the default of 300000 (5 minutes) is sufficient. Increase it only if your processing logic is exceptionally slow.

enable.auto.commit

Controls whether the consumer automatically commits offsets at a fixed interval.

auto.commit.interval.ms

Interval for automatic offset commits when enable.auto.commit is true.

A shorter interval reduces the window for duplicate messages after a crash but increases the number of commit requests to the broker.

auto.offset.reset

Determines what happens when the consumer has no committed offset or the committed offset is invalid (e.g., the offset has been deleted due to retention policies).

Recommended value: latest. Using earliest causes the consumer to reprocess all retained messages whenever an invalid offset is encountered, which can lead to duplicate processing and a spike in consumer lag.

If your application handles offset management manually (e.g., storing offsets in an external database), set this to none.