Kafka Without ZooKeeper: The KRaft Migration Guide for Teams Running Kafka in Production

In March 2025, Apache Kafka 4.0 shipped with ZooKeeper support completely removed. Not deprecated. Removed. If you try to start a Kafka 4.0 broker configured against a ZooKeeper ensemble, it does not start. Full stop.

I have been running Kafka clusters in production for twenty years, back when it was a LinkedIn internal project and the documentation fit on a single page. ZooKeeper was always the thing you tolerated. You monitored two separate systems, sized two separate clusters, dealt with ZooKeeper’s leader election quorums, debugged split-brain scenarios, and explained to every new team member why a messaging system needed a separate distributed coordination service before it would accept a single message. It was fine. You got used to it. But it was never good.

KRaft (Kafka Raft Metadata) fixes this. It is not a cosmetic change. It is a fundamental architectural shift in how Kafka manages metadata, and it changes the operational playbook for every team running Kafka in production. If your cluster is still on ZooKeeper mode in Kafka 3.x, you cannot upgrade to 4.0 without migrating first. That migration needs to happen now, not later, because Kafka 3.x is approaching end-of-life and you do not want to be the team debugging ZooKeeper in 2026.

This guide covers what ZooKeeper was actually doing, how KRaft replaces it, the migration paths available to you, and what changes in day-to-day Kafka operations after the switch.

What ZooKeeper Was Actually Doing in Kafka

To understand why KRaft matters, you need to understand what ZooKeeper was doing in the first place. Most engineers know it vaguely as “the coordination service,” but the specifics matter for the migration.

ZooKeeper was responsible for four things in a Kafka cluster:

Controller election. One Kafka broker acts as the controller, responsible for partition leadership assignments across the cluster. ZooKeeper held an ephemeral node that brokers watched. When the controller died, ZooKeeper ran the election to pick a new one. This worked, but controller failover took tens of seconds because the new controller had to read the entire cluster metadata from ZooKeeper before it could serve requests.

Broker registration. Each broker registered itself with ZooKeeper on startup and maintained a heartbeat. ZooKeeper provided the list of live brokers to the controller. If a broker died and its ZooKeeper session expired, the controller kicked in to reassign its partitions.

Topic and partition metadata. ZooKeeper stored all topic configurations, partition counts, replica assignments, and ISR (in-sync replica) lists. Every metadata change had to be written to ZooKeeper before Kafka acknowledged it.

Consumer group coordination. This actually moved out of ZooKeeper years ago, into Kafka’s internal __consumer_offsets topic. But older clients still use ZooKeeper for this, and the confusion persists.

The problem with this design is that ZooKeeper has its own consistency model and its own scaling limits. In large clusters, the ZooKeeper state grew to hundreds of megabytes, which made controller failover and cluster restarts slow. At Kafka’s design scale, ZooKeeper became a ceiling. Confluent documented that a single ZooKeeper-based Kafka cluster was limited to roughly 200,000 partitions before things got unstable. KRaft pushes that limit to millions of partitions, and the controller failover that took seconds now takes milliseconds.

The KRaft Architecture: How It Actually Works

KRaft replaces ZooKeeper with a Raft-based consensus protocol running inside Kafka brokers themselves. Instead of a separate ZooKeeper ensemble, you designate a set of brokers as “controllers” (three in most production setups, five for larger clusters), and these controllers form a Raft quorum. The active controller is the Raft leader. When the leader fails, Raft elects a new leader from the quorum, and because the new leader already has all the metadata in its local log, failover is nearly instantaneous.

Metadata in KRaft is stored in a special internal Kafka topic called @metadata. Every change to cluster state (broker registration, topic creation, partition leadership) is written as a record to this topic. Regular brokers act as followers of the metadata topic, pulling updates from the active controller. This is a much cleaner model: Kafka’s metadata replication now uses the same mechanism as Kafka’s data replication, which means fewer moving parts and better understood failure modes.

There are two deployment modes for KRaft:

Combined mode. Each broker is both a broker and a controller. This simplifies deployment for smaller clusters (three to five nodes) and is what most managed services use. Every node participates in both the Raft quorum and data serving.

Isolated mode. Dedicated controller nodes that do not serve Kafka client traffic. This is what you want for large clusters (50+ brokers) because it prevents controller work from competing with produce/consume workloads. The controller nodes are sized smaller (they do not need the heap for message buffering), but they must be reliable.

For most teams, combined mode is fine for clusters up to around 20 brokers. Beyond that, dedicated controllers earn their keep.

The Migration Path: Getting from ZooKeeper to KRaft

This is where the pain lives. Kafka 4.0 does not support ZooKeeper mode, so there is no “upgrade in place” from a ZooKeeper cluster to Kafka 4.0. The migration must happen on Kafka 3.x, specifically 3.7 or later where the migration tooling was declared production-ready.

Aiven, which operates tens of thousands of Kafka clusters as a managed service, migrated 15,000 servers from ZooKeeper to KRaft over three months with zero downtime. Most of that was automated. But the tooling exists for self-managed clusters too.

The officially supported migration path for a running ZooKeeper cluster is:

Step 1: Upgrade to Kafka 3.7 or 3.9 (recommended 3.9). The migration tooling improved significantly between 3.6 and 3.9. If you are on 3.5 or earlier, upgrade first. 3.9 is the last major 3.x release and will have the longest support runway.

Step 2: Deploy KRaft controller nodes. Add three (or five) dedicated KRaft controllers to your cluster. These brokers have process.roles=controller set in their configuration and controller.quorum.voters pointing to their own addresses. They do not serve producer/consumer traffic.

Step 3: Run the migration tool. The kafka-storage.sh tool with the format command initializes the controller cluster. Then you use kafka-metadata-migration.sh to initiate the migration, which copies all existing ZooKeeper metadata into the KRaft metadata log. This process is non-destructive; your ZooKeeper cluster stays running throughout.

Step 4: Migrate brokers. One broker at a time, you roll restart each broker with its process.roles=broker configuration pointing at the KRaft controllers instead of ZooKeeper. During this rolling restart, the migrating broker switches from reading ZooKeeper metadata to reading from the KRaft @metadata topic. Your producers and consumers see nothing.

Step 5: Decommission ZooKeeper. Once all brokers are migrated, you verify cluster health, confirm no ZooKeeper connections remain, and tear down the ZooKeeper ensemble. This is the moment you have been waiting for since 2013.

The entire migration for a reasonably sized cluster (20-30 brokers, a few thousand partitions) typically takes a few hours of rolling restarts. The actual migration step itself is fast. The slow part is the rolling restart cadence and watching your ISR lists stabilize between each broker restart.

One important caveat: your Kafka clients do not need to change. The broker API that clients talk to is unchanged. Producers, consumers, Kafka Streams apps, Connect workers, all of these keep working. The migration is entirely on the server side.

What Changes in Day-to-Day Operations

Once you are on KRaft, several things change in your operational workflow, mostly for the better.

ZooKeeper tooling is gone. The zookeeper-shell.sh script that you used to inspect cluster state, check topic configurations, and debug partition issues is removed in Kafka 4.0. The replacement is the Kafka metadata shell: kafka-metadata-shell.sh --snapshot /path/to/metadata/snapshot. You navigate it like a filesystem. Most things you did in ZooKeeper shell are now accessible via kafka-topics.sh, kafka-configs.sh, and kafka-metadata-shell.sh. Some teams have a learned reflex to reach for ZooKeeper shell when something is wrong; that needs to get retrained.

Controller failover is genuinely fast now. With ZooKeeper, controller failover involved a new controller loading the entire state from ZooKeeper, which could take 30 seconds or more on large clusters. With KRaft, the new controller already has all the metadata in its local log. Failover happens in under a second. This changes how you design availability around controller failures: what used to be a meaningful service disruption is now a blip.

Partition limits scale dramatically. The hard ceiling around 200,000 partitions in ZooKeeper mode is gone. KRaft clusters handle millions of partitions. If you have been artificially limiting your topic design to avoid hitting this ceiling, that constraint is lifted. Though I will say: just because you can have 10 million partitions does not mean you should. The right partition count is still driven by your throughput requirements and consumer parallelism, not what the coordinator can handle.

Kafka’s admin API covers everything. With ZooKeeper, some administrative operations had to go directly to ZooKeeper because the Kafka admin API did not expose them. Topic deletion in older versions, for example, required ZooKeeper intervention when the broker was unresponsive. In KRaft mode, the admin API is the single source of truth for all cluster operations. This makes automation more reliable.

Monitoring changes. Your ZooKeeper-specific metrics are gone. The JMX metrics you tracked for ZooKeeper connection counts and session durations are no longer relevant. In their place, you monitor the KRaft quorum directly: kafka.controller:type=KafkaController,name=ActiveControllerCount (should be 1 in a healthy cluster), kafka.controller:type=KafkaController,name=LastCommittedRecordOffset (tracks metadata log progress), and the quorum state metrics that tell you which node is the active leader and how far behind the followers are. Update your Grafana dashboards accordingly.

Kubernetes and KRaft: Strimzi and the Operator Model

If you run Kafka on Kubernetes, the picture is largely the same but with operator-specific nuances. The Strimzi Kafka operator fully supports KRaft mode as of Strimzi 0.40 (early 2024) and considers it stable for production. Kafka 4.0 deployments via Strimzi are KRaft-only, since the underlying Kafka itself removed ZooKeeper support.

The Strimzi Kafka CRD changed to reflect this. You now specify spec.kafka.metadataVersion instead of managing ZooKeeper configuration blocks. The operator handles the controller quorum setup automatically. For combined mode deployments (the default for clusters under around 10 brokers), Strimzi configures all nodes as both brokers and controllers.

One thing to be aware of with Strimzi and KRaft: the KafkaRebalance CRD for Cruise Control still works, but some features that previously inspected ZooKeeper state now go through the Kafka admin API. Strimzi 0.43 and later handle this transparently.

For teams using the Kubernetes-native Kafka operator from Confluent (the Confluent for Kubernetes offering), KRaft support arrived in a similar timeframe. The migration tooling is wrapped in the operator’s day-2 operations playbook, which automates the rolling migration across controller and broker pods.

Production Readiness: What to Check Before You Migrate

Before you start the migration on a production cluster, work through this list.

Client compatibility. Your Kafka clients need to be on a reasonably modern version. Clients from before Kafka 2.0 sometimes use ZooKeeper directly for producer metadata or consumer group coordination. Check your producers and consumers. Kafka Streams apps are fine as long as they are on Kafka 2.8 or later. Kafka Connect is fine on 2.6 or later. Legacy clients that go directly to ZooKeeper for group coordination will break, but honestly, if you have those in production in 2025, you have bigger problems.

MirrorMaker 2. If you use MirrorMaker 2 for replication between clusters (common for disaster recovery and geographic replication), the MirrorMaker 2 cluster itself needs to be on Kafka 3.7 or later before you migrate either the source or target cluster. MirrorMaker 2 is a Kafka Connect cluster under the hood, so the client compatibility rules apply.

Schema Registry. Confluent Schema Registry and its open-source equivalents do not connect directly to ZooKeeper; they use Kafka topics for storage. Migration is transparent to Schema Registry. The Kafka Schema Registry and Avro/Protobuf setup you have in production does not need to change.

Kafka Connect. Connect workers run as Kafka clients; they do not need ZooKeeper access. Your connectors keep running through the migration. This includes Change Data Capture connectors running Debezium, which use Kafka Connect as their runtime.

Monitoring infrastructure. As mentioned, your ZooKeeper monitoring needs to be replaced with KRaft quorum monitoring. Do not decommission your old dashboards until you have built and validated the new ones. Running both in parallel for a week after migration is reasonable.

Backup the ZooKeeper data directory first. The KRaft migration is non-destructive, but take a ZooKeeper snapshot before you start. If something goes sideways, you want a known-good ZooKeeper state to restore from. This is the equivalent of taking a database backup before a schema migration: you hope you never need it, but you never skip it.

When to Stay on Kafka 3.x (Briefly)

I want to be direct about this: there is no good reason to stay on ZooKeeper mode at this point. The migration tooling works. Managed Kafka services (MSK, Confluent Cloud, Aiven, Redpanda) have all migrated their infrastructure. If you are on a managed service, you may already be on KRaft without realizing it.

The one scenario where I would say take your time is if you are running a very large cluster (hundreds of brokers, millions of partitions) with unusual configurations and heavy use of less-common Kafka features. In that case, doing a full end-to-end test migration on a staging environment that mirrors production before touching production is worth the extra few weeks. The migration itself is reliable; your environment specifics may not be.

If you are evaluating whether to stay with Kafka or move to an alternative, that is a different question. Kafka alternatives like Redpanda, AutoMQ, and WarpStream have different operational tradeoffs. Some of them are architecturally simpler because they started without the ZooKeeper constraint. But for teams already invested in Kafka’s ecosystem, the KRaft migration is the right path, not a platform switch.

What Kafka 4.0 Changes Beyond KRaft

KRaft is the headline, but Kafka 4.0 has other changes worth knowing about.

Queue semantics (early preview). Kafka 4.0 introduces support for queue-style consumption where multiple consumers in the same group share a single partition. Historically, Kafka’s parallelism model required one consumer per partition, which meant that to scale consumption you had to increase partition count. Queue semantics change this. It is in early preview in 4.0 and not production-ready yet, but it is a significant architectural addition that will affect how you design topics.

Faster consumer group rebalances. The new “incremental cooperative rebalancing” is now the default, replacing the stop-the-world rebalance that every Kafka developer has cursed at some point. With incremental cooperative rebalancing, consumers that are not affected by the rebalance keep consuming during it. This materially reduces the rebalance impact in large consumer groups.

Improved tiered storage. Kafka’s tiered storage feature (keeping old log segments in object storage while keeping recent data on local SSDs) is more mature in 4.0. This is particularly relevant for data engineering pipelines where you need long retention but cannot afford to provision SSDs for months of data. The economics improve significantly when you can tier cold data to S3 or GCS.

Client protocol improvements. The Kafka protocol version in 4.0 drops support for older protocol versions that were kept for backward compatibility. If you have clients from the Kafka 0.x era, they stop working with 4.0 brokers. This is almost certainly not your problem in 2025, but it is worth a quick audit of your client versions.

The Operational Reality After Migration

I have been through this migration on a handful of clusters over the past year, both managing it myself and advising teams going through it. The honest summary: it is less scary than it looks from the outside.

The rolling migration approach means your producers and consumers keep running throughout. The ZooKeeper ensemble stays running and healthy until the very end, so if anything goes wrong, you have a rollback path. The Kafka 3.9 migration tooling is solid. Confluent’s documentation on this is unusually good.

The parts that take the most time are not the migration itself but the preparation: inventorying your clients, validating your monitoring, making sure your team understands the new operational model. Budget two weeks of preparation for every day of actual migration work.

After migration, the cluster is genuinely simpler to operate. Fewer things to monitor, no ZooKeeper ensemble to keep healthy, no debugging split-brain scenarios between Kafka and ZooKeeper. Controller failures that previously created meaningful outage windows now resolve in under a second.

For teams building stream processing systems with Apache Flink, the KRaft migration is transparent at the Flink layer, but it does remove one category of production incidents you previously had to plan for: ZooKeeper-induced Kafka controller storms that caused checkpoint failures and job restarts.

The Bottom Line

Kafka 4.0 with KRaft is a better system than Kafka 3.x with ZooKeeper. Not marginally better. Architecturally cleaner, operationally simpler, higher partition limits, faster failover. The migration path is well-documented and has been proven at scale by managed service providers running hundreds of thousands of clusters.

If you are still on ZooKeeper mode, the time to migrate is Kafka 3.9 on your own terms, not when Kafka 3.x reaches end-of-life and you are forced into it under time pressure. The tooling works. The operational model after migration is better. There is no upside to waiting.

Twenty years of running distributed systems has taught me that the right time to do maintenance you know you need is before it becomes urgent. The ZooKeeper sunset is not a surprise; Kafka announced this direction in 2020 with KIP-500. If you have been deferring the migration, the deferral period is over.

Migrate to KRaft. Decommission your ZooKeeper ensemble. Stop explaining to new engineers why your messaging system needs a separate distributed coordination service.