II · 05 · Performance Tuning: Throughput ⇄ Latency

Source: Apache Kafka 4.4.0-SNAPSHOT (git 04bfe7d, 2026-06-15), KRaft mode. Operational guidance grounded in source code and cited benchmarks.

Kafka tuning is not a search for one fast configuration; it is the deliberate choice of where on three axes, throughput, latency, and durability, you want to sit. Almost every knob trades one against another, and the trade is mechanical: batching amortizes per-request cost at the price of a fixed latency tax; acks=all buys durability with a replication round-trip; compression buys network and disk savings with producer CPU. This chapter maps each dial to the exact source mechanism that makes it a dial, gives you the compiled-in default and the file it lives in, and then breaks the end-to-end latency budget into the five broker stages you can actually measure. The cardinal skill is reading the signal, the request-queue/local/remote breakdown and the two idle-ratio gauges, so that you tune the stage that is actually slow rather than the one folklore blames.

The three-axis model: what you are really trading

Hold this picture before touching any config. A produce request's success has three independent costs. Latency is how long the producer waits for the ack. Throughput is how many bytes per second the pipeline sustains. Durability is how many simultaneous failures the data survives. You optimize for at most two; the third is the bill. Batching raises throughput and (under load) can even lower latency, but raises latency under light load. Compression raises effective throughput and cuts cost but spends CPU. acks=all raises durability but adds a remote round-trip to latency. Confluent's reference benchmark demonstrates the happy region, roughly 605 MB/s peak and p99 5 ms at 200 MB/s on i3en.2xlarge, RF=3, acks=all, fsync off (Confluent "Kafka Performance", empirical, 1 KB messages, reads served from page cache), but note the words "AND, not maxed simultaneously": you do not get peak throughput and minimum latency at the same instant.

Producer: the batching dial

The producer's dominant lever is batching, and its mechanism is the RecordAccumulator, a per-partition deque of ProducerBatch objects (clients/.../producer/internals/RecordAccumulator.java:76,78 hold batchSize and lingerMs; the class is documented at line 102 as trading "some latency for potentially better throughput due to more batching"). A batch becomes ready to send when either it fills to batch.size or linger.ms elapses since the first record landed (the ready/allBatchesFull logic, RecordAccumulator.java:416), whichever comes first. That single OR is the whole dial. The sender thread then drains all ready batches across partitions into per-broker requests, capped by max.request.size (default 1 MB, ProducerConfig.java:424). Part I chapter 16 · Producer Client walks the accumulator and sender threads in full.

The throughput recipe vs the latency recipe

Two named configurations bracket the dial. For maximum throughput: batch.size in the 64 KB–1 MB range, linger.ms 5–100 ms, compression.type=lz4 (or zstd when storage cost dominates), acks=all with min.insync.replicas=2 (Intel/Confluent learn page, empirical). For minimum latency: keep linger.ms at 0–1 ms, modest batch.size, and accept lower compression ratios. The historical "10–50× throughput from batching" figures are vendor roll-ups, directional, not guarantees, but the direction is real and rooted in the per-request fixed cost the accumulator amortizes.

Ordering, idempotence, and max.in.flight

The interaction between max.in.flight.requests.per.connection (5) and idempotence is a correctness boundary, not a performance one. The constant MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE = 5 (ProducerConfig.java:277) is a hard cap: the broker retains at most 5 batches per producer (aligned to ProducerStateEntry#NUM_BATCHES_TO_RETAIN, comment at line 276), so a 6th in-flight batch could be silently dropped on the broker side. The validation throws ConfigException if you set it above 5 with idempotence on (ProducerConfig.java:640). With idempotence, all five in-flight requests preserve order because the broker dedups and reorders by sequence number; without idempotence, max.in.flight > 1 plus retries can reorder records on a failed-then-retried send (ProducerConfig.java:301 doc). Part I chapter 14 · Transactions & EOS details the sequence-number mechanism.

Compression: a CPU-vs-ratio dial, applied per batch

Compression happens on the full batch (ProducerConfig.java:247 doc), which is why batching and compression compound: a bigger batch compresses better. The data travels and is stored compressed, Kafka replicates and persists the compressed batch, so a good ratio cuts produce bytes, replication bytes, disk, and consumer-fetch bytes all at once. The cost is producer CPU. Kafka 4.x exposes per-codec levels: compression.gzip.level, compression.lz4.level, compression.zstd.level (ProducerConfig.java:410-412, defaults are each codec's defaultLevel()).

Speeds and ratios: Cloudflare "Squeezing the firehose" (lzbench on ~1 MB / 600-record batches; HTTP-request data), empirical and data-dependent. Cloudflare chose Zstandard level 6, saving "hundreds of gigabits of internal traffic and terabytes of flash storage" and cancelling a hardware expansion; Trendyol chose zstd level 3 for ~70% message-size reduction (empirical).

Consumer: read batching and the processing budget

On the read side, the symmetric batching dial is fetch.min.bytes + fetch.max.wait.ms. The broker holds a fetch open until it has fetch.min.bytes available or fetch.max.wait.ms elapses (this is local-log fetch; remote/tiered fetch has its own remote.fetch.max.wait.ms, noted in ConsumerConfig.java:208). The crucial fact is the default: fetch.min.bytes=1 (ConsumerConfig.java:187) means Kafka does not batch reads by default, it answers as soon as a single byte is available, optimized for millisecond latency, not for cost. Part I chapter 09 · Fetch Path covers the broker-side delayed-fetch purgatory that implements this wait.

Consumer fetch memory is bounded two ways and you must provision RAM for the larger: fetch_memory ≈ #brokers × fetch.max.bytes, and also ≈ #partitions × max.partition.fetch.bytes (Strimzi, empirical). With defaults, a consumer subscribed to 1,000 partitions can demand up to ~1 GB of fetch buffering by the second bound, a reason not to point one consumer at thousands of partitions. Partition-count economics are in II · 03 · Partitioning.

Broker: threads, queues, and how to size them from idle ratios

A broker has two thread pools on the request path. Network threads (num.network.threads, default 3 per listener, SocketServerConfigs.java:152) read bytes off sockets and write responses; each listener except the controller listener gets its own pool (doc, line 153). I/O threads a.k.a. request handlers (num.io.threads, default 8, ServerConfigs.java:51) do the actual work, log append, fetch, index lookups, including disk I/O. Between them sits the request queue, capped by queued.max.requests (default 500, SocketServerConfigs.java:144); when it fills, network threads stop reading, that is the broker's intake backpressure. Part I chapters 06 · Network & Threading and 07 · Request Processing describe this acceptor → processor → handler pipeline in detail.

Replication parallelism and the ISR-drop window

Followers replicate by issuing fetch requests through num.replica.fetchers threads (default 1). One thread per source broker fetches all that broker's leader partitions sequentially, so a single slow or hot leader can starve the rest. When a follower falls behind, the leader drops it from the ISR after replica.lag.time.max.ms (default 30000 ms, ReplicationConfigs.java:55) without a successful fetch reaching the leader's LEO. The follower fetch itself blocks up to replica.fetch.wait.max.ms (default 500 ms, ReplicationConfigs.java:75), and the doc is explicit that this must stay below replica.lag.time.max.ms to avoid frequent ISR shrinking on low-throughput topics (line 76). Each follower fetch pulls up to replica.fetch.max.bytes per partition (default 1 MB, line 68). The durability consequences of ISR membership are the subject of Part I 08 · Replication.

Socket buffers and segment size

The bandwidth-delay product is the formula for socket buffers: buffer = link_bandwidth × round-trip latency. On a 10 Gbps link with 1 ms RTT that is ~1.25 MB, so the 100 KB default throttles a single high-BDP connection, raise both socket buffers (and the OS net.core.rmem_max/wmem_max) for cross-region replication or fat WAN consumers. log.segment.bytes is a storage-engine knob (Part I 03 · Storage Log Engine): smaller segments give finer retention and faster individual-segment recovery but multiply open file descriptors and .index mmaps, which feeds directly into the file-descriptor and vm.max_map_count limits below.

The page cache is the performance engine

Kafka's read and write speed comes from the OS page cache, not the JVM heap. Writes go to the page cache and are flushed by the kernel; reads of the log tail are served from the page cache; and the fetch path uses zero-copy sendfile to ship bytes from page cache straight to the socket without copying through user space or the heap (Part I 03 · Storage Log Engine and 09 · Fetch Path). The operational consequence is counter-intuitive: keep the JVM heap small so the OS has more RAM to cache the tail. Kafka does not need a heap above ~6 GB; on a 64 GB box with a 6 GB heap, roughly 28–30 GB is left for page cache (Confluent, empirical). Note that by default Kafka does not fsync on every write, log.flush.interval.messages defaults to Long.MAX_VALUE (ServerLogConfigs.java:101), i.e. flushing is left to the OS, which is precisely why durability under acks=all means "in the ISR's page cache," not "on disk" (see II · 06 · Durability).

The end-to-end latency budget

When a request is slow, do not guess which stage, read it. The broker decomposes every request's TotalTimeMs into five additive stages, each a JMX histogram under kafka.network:type=RequestMetrics,name=…,request={Produce|FetchConsumer|FetchFollower}. The names are compiled in at server/.../network/metrics/RequestMetrics.java:48-54:

TotalTimeMs = sum of the below

The full server-side latency for the request (RequestMetrics.java:54). Plus, when throttled, ThrottleTimeMs (line 51).

RequestQueueTimeMs

Time waiting in the request queue for a free I/O thread (RequestMetrics.java:48). High ⇒ too few num.io.threads (handler starvation).

LocalTimeMs

Time the I/O thread spends doing the work locally, log append, page-cache/disk, index (RequestMetrics.java:49). High ⇒ disk/flush latency, page-cache misses, or GC.

RemoteTimeMs

Time waiting on other brokers, for a produce with acks=all, waiting for ISR acks; for a fetch, waiting in delayed-fetch purgatory (RequestMetrics.java:50). High ⇒ slow followers / min.insync.replicas waits.

ResponseQueueTimeMs

Time the built response waits for a network thread to send it (RequestMetrics.java:52). High ⇒ too few num.network.threads.

ResponseSendTimeMs

Time actually writing the response to the socket (RequestMetrics.java:53). High ⇒ slow client/network, small socket buffers.

Two more diagnostic gauges from the same class: MessageConversionsTimeMs (line 45) flags expensive down-conversion for old clients, and TemporaryMemoryBytes (line 46) tracks transient memory for conversion/compression. Throttle time (ThrottleTimeMs, line 51) is added when a client exceeds its quota, covered in Part I 19 · Quotas; from the latency-budget view it is "the broker deliberately delaying you," not a fault.

OS-level tuning: the kernel knobs that gate Kafka

Because Kafka leans on the page cache, zero-copy, and thousands of memory-mapped index files, a handful of kernel settings gate its performance and stability. These are emergent limits, consequences of the architecture, not Kafka configs, and getting them wrong produces failures that look like Kafka bugs.

Putting it together: two reference profiles

The knobs are not independent; they compose into a posture. Two endpoints, both grounded in the defaults and mechanisms above. Treat all benchmark numbers as empirical and version/hardware-dependent, the canonical LinkedIn 2014 result is 2,024,032 rec/s (193 MB/s) with three producers and async replication, but a single durable producer at acks=-1 drops to 421,823 rec/s (40.2 MB/s), sync replication roughly halved throughput (Jay Kreps/LinkedIn, empirical). That gap is the durability tax, measured.