II · 09 · Topologies & Deployment Patterns

Source: Apache Kafka 4.4.0-SNAPSHOT (git 04bfe7d, 2026-06-15), KRaft mode. Operational guidance grounded in source code and cited benchmarks.

A Kafka cluster is a physical object: it lives in racks, in availability zones, in regions, and on a budget. The same logical cluster can be one combined node on a laptop or 150 brokers fronted by an object store spanning a continent, and the layout decisions you make at formatting time (how many controllers, how you tag racks, how many clusters, how you bridge regions) are the hardest to change later and the most expensive to get wrong. This chapter is the layout playbook: the KRaft node topology (dedicated vs combined controllers, static vs dynamic quorums), rack/multi-AZ placement and how StripedReplicaPlacer turns broker.rack into AZ-survivable partitions, the blast-radius math of single vs multi-cluster, the two ways to span regions (MirrorMaker 2 and stretch clusters), where tiered storage and an object store fit, and a concrete recommended topology for each scale tier from dev to global. Every default and limit is pinned to the source that defines it; every benchmark and cost figure is marked empirical and attributed.

The control plane: KRaft node topology

Since 4.0, ZooKeeper is gone, there is no separate consensus system to deploy. The cluster's metadata (every topic, partition, ISR, config, ACL, and broker registration) is itself a single-partition replicated log, __cluster_metadata, maintained by a Raft quorum of controller nodes. A node's job is set at startup by process.roles, which accepts broker, controller, or both (raft/.../KRaftConfigs.java:32, validated to that exact list at :73). KafkaRaftServer constructs a BrokerServer if the roles contain BrokerRole and a ControllerServer if they contain ControllerRole (core/.../KafkaRaftServer.scala:76-90), so "dedicated controller", "dedicated broker", and "combined" are all the same binary with a different one-line config. See Part I · 10 (KRaft consensus) and Part I · 11 (the controller) for the mechanism this section deploys.

Why the quorum is odd, and how many failures it tolerates

A Raft quorum makes progress only when a majority of voters is reachable and current. With n voters the majority is floor(n/2) + 1, so the cluster tolerates floor((n-1)/2) simultaneous voter failures. This is the single most important number in the control plane:

Raft commits an append only after a majority has it, so commit latency tracks the median voter's round-trip, and elections are bounded by the KRaft timeouts. A follower becomes a candidate after controller.quorum.fetch.timeout.ms (default 2000 ms) without a successful fetch from the leader, and an election that stalls is retried after controller.quorum.election.timeout.ms (default 1000 ms) with exponential backoff capped by controller.quorum.election.backoff.max.ms (default 1000 ms), all in raft/.../QuorumConfig.java:77-88, defined at lines 123-125. These are deliberately low: the class header states the design philosophy that "changing the leader of a Raft cluster [is] a relatively quick operation … the standby is a 'hot' standby, not a 'cold' one" (QuorumConfig.java:46-49). A standby controller already holds the full metadata in memory, so failover does not reload state, this is the structural reason KRaft removed the ZooKeeper-era O(partitions) controller-failover penalty (empirically: ZK controlled-shutdown of 5 brokers / 50,000 partitions fell from 6.5 min on 1.0.0 to ~20–30 s under KRaft for far larger counts, Confluent KRaft lab ~2M-partition test; empirical, version-dependent).

Dedicated controllers vs combined nodes

Because the role is just config, you choose between two physical topologies:

Defining quorum membership: static voters vs dynamic (KIP-853)

There are two ways to tell the cluster who the voters are, and you pick one at format time:

• Static voters, set controller.quorum.voters to a comma-separated list of {id}@{host}:{port} (e.g. 1@host1:9093,2@host2:9093,3@host3:9093), parsed in QuorumConfig.parseVoterConnections (QuorumConfig.java:219-257). The membership is fixed in config on every node. Simple, but changing the voter set means a coordinated config rollout and restart, and you cannot grow the quorum live.
• Dynamic voters KIP-853, leave controller.quorum.voters unset, set controller.quorum.bootstrap.servers instead, and establish the initial voter set at format time with kafka-storage.sh ... --standalone (single voter) or --initial-controllers (the voters doc spells this out at QuorumConfig.java:62-63). Voters are then added and removed at runtime via the AddRaftVoter / RemoveRaftVoter RPCs (the request schema exists at clients/.../message/AddRaftVoterRequest.json; handler state in raft/.../LeaderState.java:78-79). A new controller can even self-register when controller.quorum.auto.join.enable is true (default false, QuorumConfig.java:106-109).

One more control-plane fact you will tune at scale: a broker's liveness in the cluster is a lease. Brokers heartbeat to the controller every broker.heartbeat.interval.ms (default 2000 ms) and the lease expires after broker.session.timeout.ms (default 9000 ms) of silence, after which the controller fences the broker and reassigns leadership (KRaftConfigs.java:39-45). That ~9 s is the upper bound on how long a hard-dead broker keeps "holding" leaderships before the controller moves them, see op07 (failure modes) for the failover runbook and op08 (signals) for the metrics that page on it.

Rack & multi-AZ awareness

A 3-broker RF=3 cluster all in one availability zone survives a broker failure but not a zone failure: lose the AZ and every partition is offline. The fix is to tell Kafka where each broker physically lives, via broker.rack (a free-form string, default null; doc: "used in rack aware replication assignment for fault tolerance … Examples: RACK1, us-east-1d", server-common/.../ServerConfigs.java:92-93). Set it to the AZ id. Once every broker is tagged, the controller's StripedReplicaPlacer spreads each partition's replicas across racks so no single rack/AZ holds two replicas of the same partition.

How StripedReplicaPlacer survives an AZ loss

The placer's design header is unambiguous about priority order (metadata/.../StripedReplicaPlacer.java:36-56): spreading replicas across racks is the highest-priority goal, spreading evenly across brokers is second, and preferring unfenced brokers is third. It places replicas onto racks round-robin with a random starting offset, advancing the offset by one per partition, the source illustrates the resulting "striped" pattern directly (:84-105): for racks A, B, C with three brokers each, partition 1 → A0, B0, C0; partition 2 → B1, C1, A1; partition 3 → C2, A2, B2. Two hard constraints (not goals, placement fails if violated) bound this:

1. No two replicas of a partition on the same broker (:60-64). This caps replication factor at the broker count: a 3-node cluster cannot create an RF=4 topic, it throws InvalidReplicationFactorException (:411-417). This is an architectural constraint "from Kafka's internal design," not a tunable.
2. The leader (first replica) must be an unfenced broker (:66-69, :354-366), a brand-new partition cannot elect a leader that is down.

The cost of rack awareness: cross-AZ traffic

Rack-spreading replicas means replication traffic is, by construction, cross-AZ, and on AWS/GCP that is metered. With RF=3 across 3 AZs, every 1 GiB produced becomes ~2 GiB of cross-AZ replication (the leader ships to two followers in other AZs), plus producers land ~2/3 cross-AZ and each consumer group reads ~2/3 cross-AZ absent optimization (empirical model; Confluent / AutoMQ). Cross-AZ networking is commonly 50–90% of a self-managed cloud Kafka bill at scale (empirical; Confluent cost model, directional, AWS/GCP-centric; Azure inter-AZ has historically been free). This is the durability/cost dial: AZ survival is not free, and the lever to recover the consumer share is fetch-from-follower.

Fetch-from-follower KIP-392 lets a consumer read from a same-AZ follower instead of always the leader. The broker picks the preferred read replica via replica.selector.class (default returns the leader, ReplicationConfigs.java:140-141); set it to the rack-aware selector and tag consumers with client.rack (the client-side twin of broker.rack, clients/.../CommonClientConfigs.java:77-78) so the broker matches the consumer's AZ to a co-located replica. Aligning consumer fetch traffic this way can cut total cluster cross-AZ cost by roughly half (empirical; Grab drove consumer cross-AZ to zero, InfoQ, 2023), at the cost of up to ~hundreds of ms of added tail latency and some broker load skew. Crucially it touches only consumer reads, produce and replication cross-AZ remain. The mechanism and fetch path are in Part I · 09 (the fetch path); the full cost arithmetic and lever ordering live in op10 (cost), and quota-based tenant isolation in Part I · 19.

One cluster or many? Blast radius and tenancy

A single cluster is operationally cheap (one quorum, one set of brokers, one upgrade) but it is also a single blast radius: a runaway tenant, a bad config push, a metadata bug, or a correlated failure hits everyone on it. The largest operators deliberately cap cluster size to bound this, Pinterest holds clusters to ~200 brokers, Netflix Keystone to < 200 brokers and < 10,000 partitions, Uber federates into ~150-node clusters (empirical; LinkedIn/Pinterest/Netflix/Uber engineering posts). They run many bounded clusters rather than one giant one, then bridge with an aggregation/federation tier.

Spanning regions: MirrorMaker 2 vs stretch clusters

A single Kafka cluster's replication assumes low, stable inter-broker latency, the ISR mechanism (Part I · 08) drops a follower from the ISR after replica.lag.time.max.ms (default 30 s) of falling behind, and acks=all waits on those followers. Stretch a single cluster across a high-latency WAN and you pay that latency on every durable write and risk constant ISR churn. There are therefore two fundamentally different ways to be in more than one region.

Option A, MirrorMaker 2 (asynchronous, separate clusters)

MirrorMaker 2 KIP-382 is a set of Kafka Connect connectors (in the connect/mirror module) that asynchronously copy data between independent clusters. MirrorSourceConnector consumes from the source and produces to the target; MirrorCheckpointConnector translates and replicates consumer-group offsets; MirrorHeartbeatConnector emits heartbeats to measure end-to-end lag. The clusters stay fully independent, each has its own quorum, its own durability, and replication is best-effort-forward, so the target trails the source by the replication lag (no synchronous coupling, no WAN penalty on the producer). See Part I · 21 (Kafka Connect & MM2) for the connector internals.

Key source-grounded defaults that shape an MM2 deployment:

Topic renaming prevents replication cycles. The DefaultReplicationPolicy prefixes every mirrored topic with the source cluster's alias and a separator (default ".", connect/mirror-client/.../DefaultReplicationPolicy.java:39): formatRemoteTopic(alias, topic) = alias + separator + topic (:65-66). So topic orders from a cluster aliased us-east appears on the target as us-east.orders. This makes the origin visible in the name and lets MM2 detect already-mirrored topics, so in an active/active pair A↔B the topic orders on A becomes us-east.orders on B and is not mirrored back, breaking the loop. Consumers that want both local and remote data subscribe to a pattern (e.g. .*orders).

Active/passive vs active/active. In active/passive, one region is primary; MM2 one-way mirrors to a standby that runs no producers until failover. Simple, no conflict resolution; the standby trails by replication lag (RPO > 0). In active/active, both regions take writes and mirror to each other (the prefix scheme prevents loops); you get low local latency everywhere but must design for the fact that there is no global ordering across regions and no automatic conflict resolution, keep keys region-affine or make consumers idempotent. Uber documented the operational reality: legacy MirrorMaker (MM1) rebalance storms caused 5–10 min replication stalls and ~weekly outages, which is why MM2's Connect-based, statically-assigned model (and uReplicator-style designs) replaced it (empirical; Uber engineering). MM1's flaws are not MM2's behavior.

Option B, stretch cluster (synchronous, one cluster across nearby AZs)

A stretch cluster is a single logical cluster whose brokers and controllers span multiple AZs in the same region (or very-low-latency nearby DCs), tagged by broker.rack so StripedReplicaPlacer spreads replicas across them. Because it is one cluster, acks=all + min.insync.replicas=2 gives you synchronous durability across AZs, a committed write is already replicated into two zones, RPO = 0 for an AZ loss, and failover is automatic (the controller re-elects from the surviving in-AZ replicas). The price is that every durable write pays the inter-AZ round-trip, and a 5-voter quorum spread across AZs pays the median inter-AZ latency on every metadata commit.

Tiered storage topology

By default a partition's entire log lives on broker-local disk, so storage and compute are coupled: retention is bounded by disk, and a broker that rejoins must re-replicate cold data. Tiered storage KIP-405 (see Part I · 05) puts an object store behind the cluster, old log segments are offloaded to remote storage while only recent data stays local. It is off by default: remote.log.storage.system.enable defaults to false (storage/.../RemoteLogManagerConfig.java:58) and must be enabled cluster-wide, then per-topic. Two new retention knobs split "how long to keep locally" from total retention: log.local.retention.ms and log.local.retention.bytes, both defaulting to -2 meaning "inherit the overall retention" (RemoteLogManagerConfig.java:163,169), set them small (hours/GBs) so the broker keeps only the hot tail on disk and ships the rest to the object store.

The operational wins are concrete: Uber and Pinterest run tiered storage to decouple retention from disk (Pinterest offloads ~200 TB/day to object store) and to blunt the page-cache "catch-up tax" where historical reads evict the hot tail and spike p99 produce latency (KIP-405 tests reported ~30% p99 improvement and avoided a ~43% producer-throughput drop under historical-read load, empirical, KIP-405; workload-specific). Tiered storage is also what makes "many bounded clusters" affordable, since each cluster needs only enough local disk for its hot tail.

A recommended topology per scale tier

Putting it together, a decision guide from laptop to continent. Numbers are starting points; size partitions and brokers with the formulas in op04 (capacity) and op03 (partitioning).

Anti-patterns to avoid

• Even quorum (2 or 4 voters). Strictly worse than the next-lower odd number: same fault tolerance, larger majority, possible split-brain. Always odd.
• Combined controllers under heavy data load at scale. A produce spike can starve the metadata plane and induce elections; dedicate controllers once the cluster carries real traffic.
• All replicas in one AZ. RF=3 in a single AZ survives a broker, not a zone. Tag racks so StripedReplicaPlacer spreads across AZs.
• Asymmetric AZ broker counts. The placer prioritizes rack-spread over broker-balance, so an undersized AZ hot-spots its nodes (StripedReplicaPlacer.java:48-56). Keep AZs equal.
• Stretching one cluster across regions. WAN latency breaks ISR and the Raft quorum timeouts. Use MM2 for cross-region.
• Leaving MM2 mirrored-topic RF at the default 2 and sync.group.offsets off. Both undercut a real DR posture, raise RF to 3 and enable offset sync.
• Expecting tiered storage to cut the network bill. It cuts storage; cross-AZ replication is untouched. Reach for fetch-from-follower / diskless for networking.
• One giant cluster as the whole fleet. Cap cluster size to bound blast radius; federate with MM2, the universal pattern at hyperscale.

Next: op10 (cost) turns the cross-AZ, RF, retention, and tiered-storage choices made here into dollar figures and an ordered lever list; op11 (scaling scenarios) walks the growth path through these tiers; and op07 (failure modes) covers what each topology does when an AZ, a broker, or a controller dies.