ApsaraMQ for Kafka:Does the callback setting on the Java client affect message sending speed?

Why callbacks can slow down sending

The Kafka Java producer runs callbacks on its internal I/O thread. If a callback performs time-consuming work -- such as writing to a database or making an HTTP call -- it blocks the I/O thread and prevents the producer from sending new messages until the callback returns.

Two factors determine the severity of the impact:

• Callback processing time. The longer each callback takes, the longer the I/O thread is blocked. The producer cannot send additional messages during this time.

• max.in.flight.requests.per.connection. This parameter controls how many unacknowledged requests the producer can send on a single connection before it stops and waits. Before a blocking callback completes, the producer can send at most this many additional requests. Once the limit is reached, sending stalls until callbacks finish and free up in-flight slots.

Keep callbacks fast

To avoid degrading throughput:

• Offload heavy work to a separate thread. Process callback results asynchronously in a dedicated thread or thread pool instead of doing the work inline.

    ExecutorService callbackExecutor = Executors.newFixedThreadPool(4);
  
    producer.send(record, (metadata, exception) -> {
        // Hand off to a separate thread to keep the I/O thread unblocked
        callbackExecutor.submit(() -> {
            if (exception != null) {
                logger.error("Send failed for topic {}", metadata.topic(), exception);
            } else {
                // Perform time-consuming processing here
                persistOffset(metadata.topic(), metadata.partition(), metadata.offset());
            }
        });
    });

• Batch callback processing. Accumulate a specific number of ACKs before processing them together, rather than acting on each one individually.

    AtomicInteger ackCount = new AtomicInteger(0);
    int batchThreshold = 100;
  
    producer.send(record, (metadata, exception) -> {
        if (ackCount.incrementAndGet() >= batchThreshold) {
            ackCount.set(0);
            // Process the accumulated batch
            flushMetrics();
        }
    });

• Keep inline callbacks lightweight. Logging or incrementing a counter is fine. Avoid network calls, file I/O, or any operation that could block.

    // Lightweight callback -- safe to run on the I/O thread
    producer.send(record, (metadata, exception) -> {
        if (exception != null) {
            logger.error("Send failed", exception);
        }
    });