II · 01 · Configuration: The Tuning Surface

Source: Apache Kafka 4.4.0-SNAPSHOT (git 04bfe7d, 2026-06-15), KRaft mode. Operational guidance grounded in source code and cited benchmarks.

Configuration is the operator's control surface, the dials through which you steer durability, throughput, latency, and cost without recompiling a broker. But "configuration" in Kafka is not one flat namespace. It is four overlapping layers, static broker, dynamic cluster-wide, dynamic per-broker, per-topic, plus the client side, each resolved by a precise precedence, and each backed by a ConfigDef that hard-codes a type, a default, a validator, and an importance. This chapter teaches you to read a default straight from the source (so you never trust a stale blog), shows exactly which knobs apply live versus need a restart and why (the dynamic-update path through the metadata log), and walks the ~20 broker and topic knobs that actually move the needle, each with its default, its dynamic-ness, and the concrete mechanism it tunes. The cardinal rule throughout: every recommendation carries its WHY, rooted in a real mechanism in the code, not folklore.

The anatomy of a config: ConfigDef.define()

Every Kafka configuration, broker, topic, producer, consumer, is declared exactly once through a ConfigDef. A definition binds a key, a type, a default, an optional validator (range / enum / custom), an importance, and documentation. This is the single source of truth: when this manual cites a default, it is reading the literal argument passed to .define(...).

Take the replication ConfigDef in server/src/main/java/org/apache/kafka/server/config/ReplicationConfigs.java:154. Every column below comes from one .define() call:

Resolution at runtime is mechanically simple, and there is no hidden fallback chain at this layer. The AbstractConfig constructor calls definition.parse(originals) (clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java:118); parse() walks every defined key, and if the user supplied no value it injects key.defaultValue (clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:123). Thereafter AbstractConfig.get(key) simply returns whatever sits in the resolved values map and records the access for unused-config logging (AbstractConfig.java:174). The default you read in source is the default you get.

Five sentinel conventions recur when reading defaults, and misreading them causes real incidents:

• NO_DEFAULT_VALUE (ConfigDef.java:91), a required config with no default; absence is a hard startup error (e.g. serializer classes).
• Long.MAX_VALUE as default = "effectively disabled / unbounded." Examples: log.flush.interval.messages = Long.MAX_VALUE (never flush by count, ServerLogConfigs.java:101) and the byte-rate quotas (no quota until you set one).
• -1 = "no limit / unbounded by this dimension." log.retention.bytes = -1 (no size cap, only time, ServerLogConfigs.java:81).
• -2 = "derive from the parent config." local.retention.ms = -2 derives from retention.ms for tiered topics (LogConfig.java:146).
• null default with a secondary key = a fallback synonym exists. log.retention.ms is null and falls back to log.retention.minutes then log.retention.hours (LogConfig.java:168–170); the synonym precedence is resolved in DynamicBrokerConfig.brokerConfigSynonyms (server/.../config/DynamicBrokerConfig.java:97).

The four layers and their precedence

A broker's effective configuration is assembled from four sources in a strict order. The authoritative resolution is DynamicBrokerConfig.updateCurrentConfig() (core/src/main/scala/kafka/server/DynamicBrokerConfig.scala:429): it starts from staticBrokerConfigs, overlays cluster-wide dynamic defaults, then overlays per-broker dynamic overrides, later layers win. For log/topic configs a fifth, most-specific layer applies on top: the per-topic override carried in each partition's LogConfig.

Two subtleties that bite operators:

• The layering order flips with the scope you are writing. When you set a per-broker value, the resolver lays down cluster-defaults first then your per-broker override (per-broker wins). When you set a cluster-wide default, it lays the new defaults under any existing per-broker dynamic values (per-broker still wins), see validatedKafkaProps at DynamicBrokerConfig.scala:353–360. Net invariant: a per-broker dynamic override always shadows the cluster-wide default. If a cluster-wide change "didn't take" on one broker, suspect a leftover per-broker override.
• Topic configs are synonyms of broker log.* configs. cleanup.policy (topic) ↔ log.cleanup.policy (broker); retention.ms ↔ log.retention.ms; segment.bytes ↔ log.segment.bytes; while min.insync.replicas carries the same name in both layers. The mapping is ServerTopicConfigSynonyms.TOPIC_CONFIG_SYNONYMS, surfaced through the "Server Default Property" header in LogConfig.java:117. The broker value is the cluster default; a topic that does not override inherits it.

HARD vs SOFT vs EMERGENT limits

This manual distinguishes three kinds of bound, and configuration sits across all three. Knowing which kind you face tells you whether a number is a dial, a wall, or a consequence.

Dynamic vs restart-required: the update path

Whether a config is alterable live is not a matter of opinion, it is membership in a compiled set. DynamicBrokerConfig.ALL_DYNAMIC_CONFIGS (server/.../config/DynamicBrokerConfig.java:70) is the union of every subsystem's reconfigurable list; anything outside it is rejected for dynamic update by validateConfigs with "Cannot update these configs dynamically" (DynamicBrokerConfig.java:130–139). A second set, PER_BROKER_CONFIGS (line 62), records which of those must be scoped to a broker id; dynamicConfigUpdateModes() (line 195) maps each dynamic key to "per-broker" or "cluster-wide", exactly what kafka-configs.sh reports.

The mechanism that makes live reconfiguration safe is the two-phase apply in processReconfiguration (DynamicBrokerConfig.scala:447). First a validate pass builds a candidate KafkaConfig and calls validateReconfiguration on every affected reconfigurable; if any throws, the whole update is aborted and nothing changes. Only then does the apply pass swap in the new config and call reconfigure(oldConfig, newConfig). This is why a bad dynamic value is refused atomically rather than half-applied.

The reconfigurable subsystems and the configs they own:

The ~20 knobs that actually matter

Most of Kafka's hundreds of configs are leave-them-alone. The following are the ones an operator genuinely tunes. Every default below is read from the source cited; every "tunes" column cross-links the Part I chapter that explains the mechanism. D? = dynamically alterable.

Durability & replication

Storage, retention & cleanup

Message size & timestamps

Throughput, threads & sockets

The flush configs, and why you almost never set them

Operators new to Kafka reach for log.flush.interval.messages / log.flush.interval.ms expecting they control durability. They do not, by default, and that is deliberate.

Quotas, the throttle dials

Quotas are dynamic, client-entity-scoped configs stored like any other dynamic config. The four override keys (server-common/.../config/QuotaConfig.java:89–92):

producer_byte_rate

Bytes/sec a (user, client-id) may produce; default unlimited (Long.MAX_VALUE). Over-quota requests are delayed, not failed.

consumer_byte_rate

Bytes/sec a client may fetch; same delay mechanism.

request_percentage

Cap on the fraction of broker request-handler + network time a client may consume, protects threads from a single noisy client.

controller_mutation_rate

Rate of topic create/delete/partition-add mutations a client may issue, guards the controller from metadata storms.

The quota window is shaped by quota.window.num (11 samples) × quota.window.size.seconds (1 s) (QuotaConfig.java:42,52). Mechanism and the throttling math: 19 · Quotas; operational use: II · 10 · Cost and II · 11 · Scaling Scenarios.

Client-side configs that change broker behaviour

Three producer/consumer defaults are worth pinning because operators misquote them and because they interact with broker-side guarantees. All are read from the client ConfigDefs.

Empirical tuning baselines (mark these as benchmarks, not guarantees)

The source gives you defaults and mechanisms; it cannot give you "good" values for your hardware. These are the most-cited empirical starting points from the reference, directional, version- and workload-dependent, never SLAs:

• JVM heap ~6 GB, rest to page cache. Kafka does not benefit from a large heap; on a 64 GB box a 6 GB heap leaves ~28–30 GB for the page cache that actually serves reads/writes (Confluent). Larger heaps risk multi-100ms G1 pauses that drop ISRs (Conduktor).
• File descriptors ≥ 100,000 per broker; production brokers commonly hold >30k open handles. FDs ≈ (partitions × partition-size ÷ segment.bytes) + connections (Jun Rao / Confluent). Lowering segment.bytes raises FD and mmap pressure.
• vm.max_map_count ≥ 262144; the Linux default (~65,530) caps a KRaft broker near ~32k partitions because each partition uses ~2 mmap areas (Instaclustr, empirical, version-dependent).
• vm.swappiness=1, not 0; 0 forbids swap and removes the OOM safety net (Cloudera/Confluent).
• num.io.threads ≈ 8 × data disks, bounded by cores and disk bandwidth; num.network.threads 8–12 on busy brokers; num.replica.fetchers 4–8 when followers lag (Strimzi/Confluent).
• Throughput recipe (Intel/Confluent, illustrative): batch.size 32 KB–1 MB, linger.ms 5–100, acks=all, min.insync.replicas=2, compression.type=lz4 (or zstd for storage density). Reference baseline: ~605 MB/s peak; p99 5 ms at 200 MB/s on i3en, fsync off (Confluent OpenMessaging), a specific test, not a promise.

Operating the configuration surface

A short decision tree for "I need to change a config in production":

From here, II · 02 · Limits takes the HARD/SOFT/EMERGENT framing into the concrete ceilings (partition counts, request sizes, connection limits); II · 03 · Partitioning handles the one config you can't easily undo; and II · 06 · Durability turns the durability triad above into a full guarantee model. Everything in this manual ultimately routes back through this tuning surface.