II · 02 · Limits & Boundaries: Hard, Soft & Emergent

Source: Apache Kafka 4.4.0-SNAPSHOT (git 04bfe7d, 2026-06-15), KRaft mode. Operational guidance grounded in source code and cited benchmarks.

Every Kafka cluster runs inside a fence of limits, and the single most common production mistake is misjudging which kind of fence you are leaning on. A HARD limit is a compiled-in constant, a number baked into a .java file that no config can move (the 2 GiB segment ceiling, the 2000-partition request cap, the 10,000-record controller batch). A SOFT limit is a config with a default, movable, but movable with consequences (max.message.bytes, queued.max.requests, max.connections). An EMERGENT limit is a consequence of resources and architecture, no single constant defines it; it falls out of mmap counts, file descriptors, RAM, replication bandwidth, and recovery time (the per-broker partition ceiling, per-topic throughput). This chapter is the definitive limits catalogue: each boundary named, classified, sourced to file:line, and, the cardinal rule, explained by the mechanism that produces it. We answer the two questions operators ask most: what is the maximum number of partitions per topic and per cluster? and is ~1 GiB/s per topic a hard limit? Both answers are "it depends, and here is exactly on what."

The three kinds of limit

Before the catalogue, internalize the taxonomy, it changes how you respond to a wall. If you hit a HARD limit you must redesign (more partitions, more topics, more brokers); tuning will never help. If you hit a SOFT limit you change a config and accept the tradeoff. If you hit an EMERGENT limit you add resources or change topology, and the "limit" moves with them.

Partitions, part 1, per topic

The blunt answer to "what is the maximum number of partitions per topic?": there is no per-topic HARD ceiling on the resulting partition count. A topic is just a name with N partition replicas distributed across brokers; the partition id is an int32, so the absolute type ceiling is ~2.1 billion partitions per topic, a number you will never reach. What you will hit first are two HARD batch limits on the controller operation that creates or grows the topic, and then the EMERGENT per-broker and per-cluster ceilings below.

The controller batch limits (HARD)

Topic creation and partition growth are controller operations: the active controller must materialize one metadata record per partition (a PartitionRecord) and append them to the __cluster_metadata log (Part I 11 · KRaft Controller). Two compiled constants bound a single request:

The check is explicit: validateTotalNumberOfPartitions() sums the requested partitions and throws PolicyViolationException("Excessively large number of partitions per request.") if the total exceeds MAX_PARTITIONS_PER_BATCH (ReplicationControlManager.java:1234–1237). The code comment states the why plainly: "Exceeding this number of topics per batch has led to out-of-memory exceptions. We use this validation to fail earlier to avoid allocating the memory" (:1214–1215). It is a memory-safety guard on the controller, validated before any allocation.

The deeper trap: you can only ever increase partitions

There is no API to reduce a topic's partition count, and this is the single most under-appreciated constraint in Kafka. Worse, increasing partitions on a keyed topic silently breaks per-key ordering: the default partitioner maps a key via hash(key) % numPartitions, so growing N remaps existing keys to different partitions (e.g. 7654321 % 4 = 1 but 7654321 % 6 = 3), Confluent docs; Arpit Bhayani (empirical). Co-partitioned Kafka Streams state stores do not follow the keys either. Treat partition count as near-immutable for any keyed topic; the accepted migration is to create a new topic at the target count and replay. See II · 03 · Partitioning for the migration runbook.

Partitions, part 2, per broker (EMERGENT)

The per-broker partition ceiling is the most important EMERGENT limit in the system, because it gates blast radius, recovery time, and how many brokers you must buy. No constant defines it. It is the minimum of four independent resource bounds, whichever you hit first wins.

Bound 1, vm.max_map_count and memory-mapped index files

Every open log segment carries memory-mapped index files. The offset index and time index are mapped via MappedByteBuffer (storage/.../log/AbstractIndex.java:72), that is the whole point of the design: the OS pages index lookups directly from the buffer cache without a syscall (AbstractIndex.java:50–58). Linux caps the number of mmap regions per process at vm.max_map_count; a partition's active segment maps its offset and time indexes, so the planning multiplier is ~2 mmap regions per partition. That yields the canonical ceiling, derived from these two inputs:

Assumption, vm.max_map_count = 65,530 regions/process

kind: OS default (Linux kernel vm.max_map_count sysctl). The maximum number of memory-map regions one process may hold. This is the value Kafka inherits unless you raise it.

Assumption, ~2 mmap regions per partition

kind: derived from Kafka source (planning figure). Each open segment has an OffsetIndex and a TimeIndex, each an AbstractIndex with its own MappedByteBuffer (AbstractIndex.java:72); the active segment that must be mapped contributes ≈2. A transaction index or retained older segments push it higher, so 2 is a floor per partition, not a guarantee.

Derivation, default mmap ceiling

partitions/broker ≈ vm.max_map_count ÷ (mmap regions per partition) = 65,530 regions ÷ 2 regions/partition ≈ 32,765 partitions/broker before mmap allocation fails (the regions unit cancels, leaving partitions), Instaclustr (empirical); ops reference §2.

Raised ceiling

Set vm.max_map_count = 262144 (or higher) to lift this bound above what RAM/FDs allow first, at 2 regions/partition that is 262,144 ÷ 2 ≈ 131,072 partitions/broker of mmap headroom. Heuristic: keep it comfortably above (open segments per broker × mmaps per segment), ops reference §3 (empirical).

Because the multiplier tracks open segments (≈2 per active partition) rather than partitions directly, the ~32,765 figure is the headroom for the active-segment maps; a broker whose partitions each retain many segments hits the ceiling at a lower partition count.

Bound 2, file descriptors (open files)

Every segment is several files on disk: .log, .index, .timeindex (and possibly .txnindex), each an open file descriptor, plus one FD per client/replication socket. A broker with thousands of partitions and multiple segments each easily runs tens of thousands of open files.

Formula

open files ≈ (#partitions) × (partition_size ÷ segment.bytes) × (files per segment) + connections, Jun Rao / Confluent (empirical). Each segment is ~3 files on disk (.log, .index, .timeindex; .txnindex adds a 4th), so segments-per-partition = retained bytes ÷ segment size, and each segment costs ~3 FDs.

Worked example (illustrative, substitute your measured values):

Assumption, partitions/broker = 2,000

kind: workload-dependent. A mid-sized broker; chosen so the result is concrete.

Assumption, retained data per partition = 20 GiB

kind: workload-dependent (retention × throughput). What a partition holds on disk after retention settles.

Assumption, segment.bytes = 1 GiB

kind: cited Kafka default (log.segment.bytes, LogConfig.java:131).

Assumption, files per segment = 3

kind: derived from on-disk layout (.log + .index + .timeindex).

Assumption, open client/replication sockets = 5,000

kind: workload-dependent. One FD per connection; illustrative.

Derivation

segments/partition = 20 GiB ÷ 1 GiB/segment = 20 segments/partition. FDs from segment files = 2,000 partitions × 20 segments/partition × 3 files/segment = 120,000 files. Plus 5,000 socket FDs ⇒ ≈ 125,000 open files, already above a 100,000 ulimit, which is why the recommendation below is a floor, not a ceiling.

Recommendation

Set the broker process nofile ulimit to ≥ 100,000 (raise further when the derivation above exceeds it). Production clusters routinely run "more than 30 thousand open file handles per broker", Jun Rao (empirical).

Bound 3, memory (heap and page cache)

Per-partition memory is small but not free: each partition carries leader/follower state, fetch-session bookkeeping, and producer-state metadata, and on the client side the producer must buffer per partition. The architecture deliberately keeps the JVM heap small (~6 GB) and hands the rest of RAM to the OS page cache, which is the real read/write accelerator via zero-copy sendfile (Part I 09 · Fetch Path). Two empirical planning rules:

• The producer client should allocate "at least a few tens of KB per partition being produced" against buffer.memory (default 32 MiB, ProducerConfig.java:401), Jun Rao (empirical).
• A large heap to chase performance is an anti-pattern, it starves the page cache and lengthens GC pauses; a 32 GB G1GC heap can incur 100–200 ms pauses that drop replicas from ISR, Conduktor (empirical).

Bound 4, recovery / replay time (the one that actually bites)

The bound operators feel in an incident is not steady-state memory, it is how long a broker takes to come back. On unclean restart the broker must replay and re-index the tail of every partition's log; on a leadership move the controller must elect a new leader per affected partition. Empirically:

• Leader election costs ~5 ms per partition (assumption, kind: empirical planning figure, Jun Rao 2015, ZK-era hardware; teach the mechanism, do not predict 2026 latency). Derivation: a broker holding 1,000 leaderships ⇒ recovery time ≈ 1,000 leaders × 5 ms/leader = 5,000 ms ≈ 5 s of unavailability for those partitions on unclean loss (the leader unit cancels, leaving ms). The conclusion, recovery time grows linearly with leaders/broker, is why blast radius, not the raw mmap ceiling, caps the practical partition count.
• Set num.recovery.threads.per.data.dir above its default of 2 (ServerLogConfigs.java:147) to parallelize startup log recovery across cores.
• Tiered storage (KIP-405) cuts recovery dramatically, a rejoining broker need not re-replicate cold data, Uber (empirical). See Part I 05 · Tiered Storage.

Partitions, part 3, per cluster (EMERGENT, transformed by KRaft)

Here is the headline answer to "what is the maximum number of partitions per cluster?": there is no compiled per-cluster limit. The ceiling is EMERGENT and was structurally rewritten by the removal of ZooKeeper (KIP-500; ZK removed in 4.0). The number you may have memorized, ~200,000/cluster, is a ZooKeeper-era figure and no longer the binding constraint.

Message and batch size limits

The maximum record-batch size is a SOFT limit set by config, but it is governed by multiple interlocking configs across the producer, broker, replica fetcher, and consumer. Setting one without the others is the classic "I raised max.message.bytes and replication/consumption broke" footgun.

Does a too-large message wedge replication?, the modern answer

Historically, if a topic's max.message.bytes exceeded replica.fetch.max.bytes, a follower could be unable to fetch the oversized batch and replication for that partition would stall, a genuine wedge. Modern Kafka closed this hole in the protocol: both replica-fetch limits are explicitly not absolute maximums. The source documents the override: "if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made" (ReplicationConfigs.java:69–72 for replica.fetch.max.bytes; :89–92 for the response cap). The broker always returns the oversized first batch so the follower can advance.

The request plane limits

Several SOFT limits and one HARD-feeling cap bound how much in-flight work a broker accepts. These are the knobs behind "broker is dropping connections" and "fetch sessions evicted."

Connection caps

Connection limits default to unbounded, they are an opt-in safety mechanism, not a default protection. This surprises operators who assume a broker self-limits.

Storage limits, the 2 GiB segment and the int64 offset

Two storage boundaries shape every capacity calculation: a HARD 2 GiB ceiling on a single segment, and an effectively unbounded offset space.

The 2 GiB segment ceiling (HARD)

A log segment's data file cannot exceed Integer.MAX_VALUE bytes, ~2 GiB (2,147,483,647 bytes). This is not an arbitrary config maximum; it is forced by a 32-bit type used for the byte position within a segment. The mechanism:

• The append path uses an int for physical position. LogSegment.append() reads the current size as int physicalPosition = log.sizeInBytes(); (storage/.../log/LogSegment.java:256) and increments it per batch (:277). An int overflows past ~2 GiB.
• The offset index entry is 8 bytes: a 4-byte relative offset + a 4-byte physical position. OffsetIndex.ENTRY_SIZE = 8 (OffsetIndex.java:56); the file format is "a 4 byte 'relative' offset and a 4 byte file location" (OffsetIndex.java:44, stored via putInt at :207–208). A 4-byte file location addresses at most ~2 GiB.
• The relative offset must also fit in an int. Before each append the segment checks canConvertToRelativeOffset() → OffsetIndex.canAppendOffset() (LogSegment.java:237–239); toRelative() returns empty if relativeOffset > Integer.MAX_VALUE (AbstractIndex.java:556–557), and the append throws LogSegmentOffsetOverflowException (LogSegment.java:258, 281–283) / IndexOffsetOverflowException (AbstractIndex.java:312–314). The log rolls a new segment rather than overflow.

The int64 offset (effectively unbounded)

While the relative offset within a segment is 32-bit, the absolute partition offset is a 64-bit signed integer. The segment's baseOffset is a long (AbstractIndex.java:63), and the index stores offsets relative to it precisely so the per-segment field can stay 4 bytes while the partition's absolute offset space remains effectively infinite. How effectively infinite? The derivation, from two stated inputs:

Assumption, int64 offset ceiling = 2⁶³ − 1 ≈ 9.2×10¹⁸ records/partition

kind: HARD type width (signed 64-bit long, AbstractIndex.java:63).

Assumption, sustained write rate = 1,000,000 records/second

kind: illustrative, substitute your measured per-partition rate. A deliberately aggressive single-partition rate, far above the ~10 MB/s planning figure, to make the bound a worst case.

Derivation

time-to-exhaust = 9.2×10¹⁸ records ÷ 10⁶ records/second ≈ 9.2×10¹² seconds (records cancel); ÷ (60×60×24×365 ≈ 3.15×10⁷ seconds/year) ≈ ~290,000 years.

So offset exhaustion is not a real-world limit; segment-position overflow (handled by rolling, above) is the only practical 32-bit boundary in the storage layer.

The headline question: is ~1 GiB/s per topic a hard limit?

No. There is no compiled constant that caps a topic's throughput at 1 GiB/s, or at any value. Per-topic throughput is the canonical EMERGENT limit, and understanding why is the most useful mental model in this chapter. The reasoning rests on four numbers, each labelled below before it is used in the derivation:

Assumption, per-partition produce ceiling p ≈ 50 MB/s (conservative ~10 MB/s)

kind: workload/hardware-dependent empirical planning figure, a single partition sustains "tens of MB/sec"; ~10 MB/s is the conservative number, tuned modern Kafka reaches 50–100+ MB/s. Jun Rao; ops reference §7 (version/hardware-dependent, substitute your measured value).

Assumption, replication factor RF = 3

kind: topic config (the durable default; leader + 2 followers).

Assumption, egress amplification = RF (≈3× the raw write per leader)

kind: derived from RF. A leader serves its own write plus (RF−1) replication copies to followers; counting the inbound write and the RF−1 outbound copies, a leader's byte budget is ≈ RF × ingress.

Assumption, topic ingress target T = 1 GiB/s ≈ 1,024 MB/s

kind: illustrative engineering target (the headline question's number; substitute your own).

1. A topic's writes are spread across its partitions, and each partition has exactly one leader broker (Part I 08 · Replication). All producer traffic for a partition lands on that one leader. So a topic's aggregate write throughput is bounded by the sum of its partition-leaders' capacities, and any single partition is bounded by its single leader.
2. The per-partition ceiling is itself emergent, set by that leader's disk write bandwidth, its NIC, and the replication it must drive, this is assumption p above.
3. Therefore topic throughput ≈ (#partitions) × (per-partition ceiling p), a product whose units cancel to a rate: e.g. 20 partitions × 50 MB/s per partition = 1,000 MB/s ≈ 1 GiB/s. It is capped by the aggregate capacity of the brokers hosting those leaders, their NICs, disks, and the replication amplification of RF.
4. Replication multiplies the cost. With RF=3, every 1 byte produced is copied to RF−1 = 2 followers, so a leader's egress (and cross-AZ bytes) ≈ (RF−1) × ingress = 2 × ingress. Plan egress at ≥2× ingress, Confluent (empirical). This is why "the NIC" is often the true per-broker ceiling, not the disk: at RF=3 the NIC carries ingress + 2× ingress ≈ 3× the raw write.

Where the limit lives: Kafka vs Kinesis

One question surfaces so often it belongs in the catalogue, because answering it exercises the whole taxonomy: "Kinesis is Kafka under the hood, so where in Kafka's source is the 10 MB/s-per-shard limit?" The premise is wrong twice, and the untangling is the point. First, Kinesis is not built on Kafka. AWS Kinesis Data Streams is an independent, proprietary implementation of the same distributed-log pattern (Part III III · 00 · The Distributed Log as a Pattern); the Kafka-on-AWS product is Amazon MSK, a separate thing. A Kinesis shard and a Kafka partition are convergent designs, not shared code. Second, the number: a Kinesis shard sustains 1 MB/s (or 1,000 records/s) write and 2 MB/s (or 2,000 records/s) read (AWS Kinesis quotas). The "10 MB" is not a per-second rate: it is the most a single GetRecords call may return (5 calls/s per shard, but sustained read stays at 2 MB/s), and separately the per-record payload ceiling (raised to ~10 MiB, served via burst capacity).

The instructive part is where each limit is defined. In Kafka, per-partition throughput is the canonical EMERGENT limit from the section above: no constant sets it; it falls out of the leader's disk, NIC, and replication budget. In Kinesis the same logical number is a hard service quota, a value AWS chose and enforces at the API edge (the ProvisionedThroughputExceededException), because in Kinesis the shard is the unit of billing, so its capacity must be a declared constant. Same abstraction, opposite home for the limit.

Durability limits and the configs that bound them

A few SOFT limits govern durability and availability; their defaults are conservative-leaning but one default in particular is a footgun. These are covered in depth in II · 06 · Durability; here is the limit-relevant summary.

Quick-reference: the limits catalogue

One table to scan in an incident. Class column: HARD (compiled), SOFT (config default), EMERGENT (resource/architecture).