II · 08 · Metrics, Signals & Observability

Source: Apache Kafka 4.4.0-SNAPSHOT (git 04bfe7d, 2026-06-15), KRaft mode. Operational guidance grounded in source code and cited benchmarks.

A Kafka broker emits hundreds of metrics; an on-call engineer needs about a dozen. This chapter separates the signal from the noise. It traces the two metric pipelines in source, the legacy Yammer/KafkaMetricsGroup path that surfaces over JMX, and the client-side Kafka Metrics now pushable to the broker via KIP-714, then enumerates the golden signals in priority order, each cited to the exact line that emits it, each annotated with what it means and when to page. The single most useful diagnostic, the per-API request-latency breakdown, gets its own dissection because the decomposition (RequestQueueTimeMs + LocalTimeMs + RemoteTimeMs + ...) localizes almost every broker-side latency incident to one subsystem. Closes with a thresholds table you can paste into an alerting rule and a 12-panel dashboard blueprint.

The two metric pipelines

Kafka has carried two metrics libraries since its early history, and a production observability stack must scrape both. The distinction matters because they have different namespaces, different export mechanisms, and different aggregation semantics.

Server-side: Yammer (Dropwizard) Metrics via JMX. Every broker/controller-internal metric, bytes, request times, ISR counts, controller state, is registered through KafkaMetricsGroup, a thin wrapper over the Yammer MetricsRegistry. The wrapper constructs the JMX MetricName by concatenating a package (the JMX domain), a type, and a name: server-common/src/main/java/org/apache/kafka/server/metrics/KafkaMetricsGroup.java:59-67 builds the string group:type=typeName,name=metricName and appends tag key/value pairs. So BytesInPerSec registered by BrokerTopicMetrics surfaces as the JMX MBean kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec. These are Gauges (instantaneous values), Meters (1/5/15-minute exponentially-weighted rates plus a count), Histograms, and Timers (a histogram of durations plus a rate). You expose them with a JMX exporter (the Prometheus JMX agent is the de-facto standard) running in-process as a Java agent.

Client-side: the org.apache.kafka.common.metrics.Metrics library. Producers, consumers, Streams, Connect, and the broker's own network layer use a second, newer metrics system (Sensors + Measurables). These live under JMX domain kafka.consumer, kafka.producer, etc. Consumer lag (records-lag-max) is here, not in the Yammer tree. Historically these were only visible on the client JVM; KIP-714 changes that (below).

The golden signals, in priority order

Map Kafka onto the four Google-SRE golden signals (latency, traffic, errors, saturation), but recognize that the most valuable broker metrics are a handful of near-binary health gauges plus one latency decomposition. The table below is the triage order, top rows page, lower rows inform. Every metric is cited to the source that registers it.

1 · ActiveControllerCount, the cluster has exactly one brain

Each controller in the KRaft quorum registers a gauge that returns active ? 1 : 0, QuorumControllerMetrics.java:108-113. The active controller (the quorum leader) reports 1; standby controllers report 0. The invariant is therefore a cluster-wide sum:

Companion controller metrics that diagnose why a controller is struggling, all from QuorumControllerMetrics.java: EventQueueTimeMs (line 48), how long events wait before the single controller thread picks them up; rising values mean the controller is the bottleneck. EventQueueProcessingTimeMs (line 50), how long each event takes to apply. LastAppliedRecordLagMs (line 60), for a standby, how far its applied state trails wall-clock; a large lag means a slow standby that would be slow to take over. TimedOutBrokerHeartbeatCount (line 62) and per-broker TimeSinceLastHeartbeatReceivedMs (line 71), brokers the controller is about to fence.

2 · OfflinePartitionsCount, the outage gauge

Emitted only by the active controller (ControllerMetadataMetrics.java:125, managed by ControllerMetadataMetricsPublisher). A partition is offline when it has no leader, so producers and consumers for it get LEADER_NOT_AVAILABLE and the partition is simply down. This is the clearest "customer is affected" signal in the whole system.

3 · UnderReplicatedPartitions, degraded but serving

A leader-side gauge: ReplicaManager.scala:241 sums partitions where isUnderReplicated, and the predicate is exactly isLeader && (replicationFactor − isr.size) > 0 (Partition.scala:232). So URP counts partitions this broker leads whose ISR has fewer members than the assignment. The data is still readable and (unless also under-min-ISR) writable; you have simply lost redundancy headroom.

4 · UnderMinIsrPartitionCount, acks=all at risk

The most important durability gauge. ReplicaManager.scala:242 counts leader partitions where isUnderMinIsr, defined as isr.size < effectiveMinIsr (Partition.scala:237-239). When this is >0, any producer using acks=all on those partitions gets NOT_ENOUGH_REPLICAS and the write is rejected, the broker refuses to acknowledge a write it cannot make durable to min.insync.replicas copies. This is the mechanism that turns a silent redundancy loss into a loud produce failure, and it is exactly why you set min.insync.replicas=2 with RF=3: see Part I · 08 and II · 06.

5 · UncleanLeaderElectionsPerSec, the data-loss counter

A Meter on the active controller (ControllerMetadataMetrics.java:143-144, type ControllerStats). It ticks whenever a partition was forced to elect an out-of-sync replica as leader because no in-sync replica was available. The new leader's log is shorter than the old committed high-watermark, so committed records past its log end are lost forever, and the partition is set to RECOVERING with ISR reset to that singleton.

Signal 6 · The request-latency breakdown, your single best diagnostic

If you instrument one thing well, make it this. For every API key, RequestMetrics (RequestMetrics.java:89-119) registers a family of biased histograms under kafka.network:type=RequestMetrics,name=<component>,request=<api>. The components are not independent measurements, they are consecutive segments of one request's lifetime, computed in Request.updateRequestMetrics (server/.../network/Request.java:325-337) as differences between timestamps captured as the request flows through the broker. TotalTimeMs is the sum of the segments:

The request= tag is the API name (Produce, Fetch, ApiVersions, …) with two important overrides built in Request.java:339-356: a Fetch is split into FetchConsumer (client reads) and FetchFollower (replication reads) based on whether the request came from a follower, so you can separate consumer-facing latency from replication latency on the same broker. Always watch Produce, FetchConsumer, and FetchFollower separately. The diagnostic map:

Signals 7–9 · Saturation and ISR churn

RequestHandlerAvgIdlePercent, are the I/O threads keeping up?

The request-handler (I/O) thread pool services every request after the network thread hands it off. Each handler thread, in its run loop, measures the time it spends blocked waiting for the next request and marks it into a shared Meter (KafkaRequestHandler.scala:115-123). The metric is a fraction in [0, 1]: 1.0 = threads are entirely idle (healthy), 0.0 = threads never wait because work is always queued (saturated). Critically, the recorded idle time is divided by the thread count (idleTime / aggregateThreads.get, line 123) so the meter reads as a per-thread average fraction regardless of pool size, the comment at lines 111-114 explains why.

NetworkProcessorAvgIdlePercent, are the network threads keeping up?

The SocketServer gauge (SocketServer.scala:120-132) averages the io-wait-ratio of every data-plane network processor, each processor's fraction of time blocked in select() waiting for socket I/O, clamped to ≤1.0 and averaged across processors (with an empty-pool fallback of 1.0). Same semantics as the handler idle ratio: closer to 1 is better. Confluent's guidance is to keep it above ~0.4; below ~0.3 raise num.network.threads (empirical). Companion gauges from the same registration block: MemoryPoolAvailable / MemoryPoolUsed (line 134-135) for the request-buffer memory pool, and ExpiredConnectionsKilledCount (line 136).

IsrShrinks / IsrExpands, replication health, second derivative of URP

Two Meters on each broker (ReplicaManager.scala:257-258). In steady state both should be ~0. A shrink fires when a follower crosses replica.lag.time.max.ms and is removed from an ISR this broker leads; an expand fires when it catches up and rejoins. The useful pattern is the imbalance: shrinks without matching expands mean replicas are leaving and not coming back (a broker is genuinely down or a follower is persistently slow). A steady drumbeat of matched shrink/expand pairs means a follower is oscillating around the lag threshold, a slow-host or GC-pause signal. FailedIsrUpdatesPerSec (line 259) ticking means the leader's attempts to commit an ISR change to the controller are failing, investigate controller health.

Purgatory, the latency-vs-load lens, not an alert

Delayed operations (a Produce waiting for acks=all, a Fetch waiting for fetch.min.bytes) park in a DelayedOperationPurgatory, which registers two gauges tagged by operation type (server-common/.../purgatory/DelayedOperationPurgatory.java:97-99): kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize,delayedOperation={Produce|Fetch|...} and NumDelayedOperations. PurgatorySize counts watched operations (including some already completed but not yet purged); NumDelayedOperations is the count still pending.

Traffic, flush, and election rate

BrokerTopicMetrics (storage/.../metrics/BrokerTopicMetrics.java:33-55) is the throughput family, available both cluster-wide (untagged) and per-topic (tagged topic=…, lazily created on first traffic, line 357-359). The capacity-planning inputs (II · 04): BytesInPerSec, BytesOutPerSec, MessagesInPerSec. Replication-specific copies exist only at the broker level: ReplicationBytesInPerSec / ReplicationBytesOutPerSec (lines 95-96, registered only when the tag set is empty), plus ReassignmentBytesInPerSec/OutPerSec for live reassignment traffic. Error meters worth watching: FailedProduceRequestsPerSec and FailedFetchRequestsPerSec (lines 39-40), and the data-quality meters InvalidMessageCrcRecordsPerSec / NoKeyCompactedTopicRecordsPerSec (lines 52-54) that flag corruption or misconfigured compacted topics. If tiered storage is on, this class also exposes the remote-copy/fetch lag and rate gauges (lines 101-119), see Part I · 05.

LogFlushRateAndTimeMs (a Timer, LogSegment.java:76-77, JMX kafka.log:type=LogFlushStats) records how long fsync takes. Rising flush time is an early disk-pressure indicator that propagates into Produce LocalTimeMs and depresses RequestHandlerAvgIdlePercent, a classic correlated trio. Note: with default settings Kafka relies on the OS page cache and does not fsync on every write (the acks=all durability boundary is "in every replica's page cache," not "on disk", a critical nuance covered in II · 06).

Consumer lag, alert on the trend, from two vantage points

Lag is the gap between the log end offset and a consumer group's committed offset. There are two ways to measure it, and a robust setup uses both.

Client-side, live: the consumer's own org.apache.kafka.common.metrics tree exposes records-lag-max (FetchMetricsRegistry.java:102, JMX group consumer-fetch-manager-metrics), the maximum lag across the partitions this consumer owns, plus per-partition records-lag / records-lag-avg (lines 135-139) and the symmetric records-lead-min (line 104) telling you how close you are to the log start (and thus to losing un-consumed data to retention). The catch: a dead consumer reports nothing, so client-side lag alone cannot detect a stalled group.

External, vantage-independent: query committed offsets via the Admin API / kafka-consumer-groups.sh --describe, or run a dedicated monitor (LinkedIn's Burrow classifies each group OK/WARN/ERR by evaluating the lag trend over a sliding window, no per-topic threshold tuning; kafka_exporter is the common Prometheus alternative). This catches the stalled-consumer case the client metric misses. Group-coordination internals in Part I · 13.

Client telemetry push, KIP-714

KIP-714 closes a long-standing observability gap: client-side metrics (producer batch sizes, consumer fetch latency, lag) historically lived only on the client JVM, invisible to the cluster operator. The broker now runs a ClientMetricsManager (server/.../ClientMetricsManager.java) that lets clients push their metrics to the broker, which forwards them to a pluggable exporter. The flow: a client sends GetTelemetrySubscriptions to learn which metrics the operator wants (configured via dynamic client.metrics resources, a subscription names metric prefixes, a push interval, and a match filter), then periodically sends compressed PushTelemetry requests; the broker decodes them (supported codecs: zstd, lz4, gzip, snappy, ClientMetricsManager.java:87-88) and hands them to a ClientTelemetryExporterPlugin.

The manager keeps a per-broker LRU cache of up to 16,384 client instances (CACHE_MAX_SIZE, line 90) with a 60-second default expiry (line 91), and instruments itself via the Kafka Metrics group client-metrics (ClientMetricsStats, lines 569-625): instance-count (active client instances), unknown-subscription-request, throttle, plugin-export + plugin-export-time-avg/max (exporter latency), and plugin-error. Watch plugin-error and plugin-export-time-max to confirm telemetry is actually flowing. Architecture context in Part I · 19.

Thresholds reference

Paste-ready alert rules. "Page" = wake someone; "Ticket" = investigate next business hour; "Watch" = dashboard only. Idle-ratio and lag thresholds are community/empirical (no compiled constant); health-gauge thresholds are structural.

The 12-panel dashboard blueprint

One screen, top-left = most urgent, reading order = triage order. Every panel maps to a metric established above so the on-call path is "panel lights up → known runbook."

Pitfalls and version caveats

• Scrape both metric trees. Yammer/JMX for broker internals; the Kafka Metrics tree (kafka.consumer.* / KIP-714 push) for client signals. Consumer lag is not in the broker JMX tree.
• RequestHandlerAvgIdlePercent has a buggy lineage. KAFKA-7295 (calculation), Datadog integrations-core (rate-vs-gauge), KIP-1207 (KRaft combined-mode anomaly where one thread pool serves broker and controller roles). Verify it reads 0–1 before alerting.
• Health gauges are controller-scoped. OfflinePartitionsCount and ActiveControllerCount must be aggregated; per-node values mislead.
• Non-zero Produce purgatory is normal under acks=all. Don't alert on raw purgatory size; use it to confirm a RemoteTimeMs diagnosis.
• min.insync.replicas is silently capped to the replica count by effectiveMinIsr (Partition.scala:246-248), a topic with RF < min.insync.replicas gives no warning and reads 0 under-min-ISR. Audit per topic.
• ZooKeeper-era metrics are gone. KRaft removed ZooKeeper in 4.0; outstanding_requests, pending_syncs, KafkaZookeeperSyncConnect no longer exist. Much published monitoring guidance predates KRaft, the controller-quorum metrics (EventQueueTimeMs, LastAppliedRecordLagMs) replace them.
• Client-side records-lag-max misses dead consumers. Pair it with an external committed-offset monitor (Burrow / kafka_exporter) that evaluates the lag trend independent of consumer liveness.
• Alert on trend for noisy signals (lag, latency) and on absolute value only for the binary health gauges. Gate pages on impact and a sustained window to avoid flapping.