Running Kafka on KRaft: What Changed and What Didn't

Dropping ZooKeeper removes an entire distributed system from your operational surface. That is the headline, and it is real. What surprised me is how much of Kafka operations is unchanged by it.

What genuinely gets simpler

You go from two clusters to one. No separate ZooKeeper ensemble to size, patch, monitor, and reason about during a network partition. Metadata lives in an internal Kafka topic, replicated by the same mechanism as everything else, which means one mental model instead of two.

Controller failover is noticeably faster, and cluster startup no longer depends on a second quorum being healthy first. For a cluster of modest size this is the difference between a fifteen-minute bootstrap runbook and a five-minute one.

What is exactly as hard as before

Partition count still determines your consumer parallelism ceiling, and you still cannot decrease it. Rebalances still pause consumption. Consumer lag is still the metric that tells you whether the system is healthy, and it still needs to be per-partition, because an aggregate lag of zero hides one stuck partition beautifully.

Latency work is unchanged too. Sub-100ms end to end came from the usual levers: linger.ms tuned against batch size, acks=all with in-sync replica counts that do not stall on one slow broker, compression chosen for the payload shape, and consumers that do their slow work outside the poll loop.

Sizing the controller quorum

Three controllers for anything you care about. They can be dedicated nodes or combined with brokers in smaller deployments, and the combined mode is fine until it isn't — the failure mode is a broker under load starving the controller thread, which shows up as slow metadata propagation rather than an obvious error.

If you start combined, write down the trigger for splitting them out. Otherwise you will make that call during an incident.

Migration is the awkward part

Moving an existing ZooKeeper cluster to KRaft is a staged process with a point of no return. It works, but it is not a rollback-friendly operation, so it belongs in a maintenance window with a tested restore path — not in a Friday deploy.

Starting fresh on KRaft is a completely different experience and I would not hesitate.

What I would tell myself before starting

Instrument the cluster before you need to. Broker-level metrics are not enough; you want per-topic, per-partition lag, request latency percentiles rather than averages, and under-replicated partition counts on a dashboard someone actually looks at.

Everything I learned the hard way was visible in metrics I had not bothered to export yet.