I have been building distributed systems for twenty years, and nothing humbles a senior engineer quite like a split-brain incident at 2 AM. I watched a three-node etcd cluster split during a network partition in 2019. One node was convinced it was the leader. Another node in a different availability zone also thought it was the leader. Both started accepting writes. By the time we noticed, Kubernetes had scheduled the same pods onto conflicting nodes and our service mesh configuration had two diverging versions of truth. The blast radius was significant.
What saved us from a complete disaster was that etcd is built on Raft. Raft’s safety guarantees meant only one of those nodes could actually commit writes to a majority of the cluster. The other one’s “writes” were phantom commits that would never replicate. Raft did its job. Our monitoring did not.
That incident crystallized something I now consider essential knowledge for anyone running distributed infrastructure: you cannot operate etcd, CockroachDB, Kafka KRaft, or Consul intelligently without understanding how Raft works. Not the academic version with formal proofs, but the production version with real node failures, network timeouts, and AZ-level outages.
The Problem Consensus Solves
Before Raft, let me explain what problem distributed consensus is actually solving, because engineers often conflate it with replication.
Replication copies data from one node to another. Consensus is about multiple nodes agreeing on the same sequence of events, even when some nodes fail or become unreachable. These are related but distinct problems. You can have replication without consensus (MySQL async replication) and you can have consensus that drives replication (Raft).
The core problem is this: in a distributed system, any node can fail at any time, and you cannot distinguish a slow node from a dead one. If you have three nodes and one becomes unreachable, does it hold the latest state that the other two need, or is it just dead and the other two should proceed? The wrong decision in either direction causes data loss or data corruption.
Before consensus algorithms, distributed systems engineers dealt with this through elaborate coordination protocols, external locking services, or by simply accepting that split-brain could happen and building application-level reconciliation. These approaches work, but they are operationally brutal.
The CAP theorem codifies the fundamental trade-off here: you can have consistency or availability during a partition, but not both. Consensus algorithms like Raft make an explicit choice for consistency. When a partition happens, the minority partition stops accepting writes. That is the right choice for coordination services.
How Raft Actually Works
Diego Ongaro designed Raft in 2013 with one explicit goal: understandability. Paxos, the older consensus algorithm, is notoriously difficult to implement correctly. Raft decomposes the problem into three independent sub-problems: leader election, log replication, and safety.
Leader Election
In a Raft cluster, exactly one node is the leader at any time. All writes go through the leader. The leader sends heartbeats to all followers on a regular interval (typically 100-200ms in production configurations). If a follower does not receive a heartbeat within an election timeout (typically 150-300ms), it assumes the leader is dead and starts an election.
The election process:
- The follower increments its current term number and transitions to “candidate” state
- It votes for itself and sends RequestVote RPCs to all other nodes
- If it receives votes from a majority (quorum), it becomes the new leader
- If another leader is elected first (with a higher term number), the candidate reverts to follower
The term number is the critical concept. Every message in Raft carries a term number. If a node receives a message with a higher term than its own, it immediately recognizes the new epoch and updates its state. This prevents old leaders from interfering after a partition heals.
Quorum is what prevents split-brain. To become leader, a candidate needs votes from a strict majority of nodes: 3 of 5, 2 of 3, 4 of 7. This means that in a partitioned cluster, only one partition can have a quorum. The minority partition cannot elect a leader. Writes stop on the minority side. Consistency is preserved.
Log Replication
Once a leader is elected, all writes go through a process called log replication:
- A client sends a write request to the leader
- The leader appends the entry to its own log (not yet committed)
- The leader sends AppendEntries RPCs to all followers in parallel
- When a majority of nodes have acknowledged the write, the leader commits the entry
- The leader responds success to the client and notifies followers of the commit in the next AppendEntries
“Committed” means the entry is permanently part of the log, guaranteed to survive any future failures as long as a quorum of nodes survives. The leader only tells the client the write succeeded after it has been committed to a majority.
This is where the write latency cost of consensus lives. A write in a single-leader relational database completes when the write hits the WAL on one node. A Raft write completes when the write has been acknowledged by a majority of nodes and flushed to disk on each. In a three-node cluster in the same data center, this adds roughly one round-trip time. Across AZs, you are adding 1-5ms. Across regions, you are adding 50-150ms per write.
For coordination data like Kubernetes cluster state, this is perfectly acceptable. For high-throughput transaction processing, it requires careful thought about data partitioning and write patterns.
Safety Guarantees
Raft provides two critical safety properties:
Election safety: at most one leader can be elected in any given term. The majority voting requirement ensures this.
Log matching: if two logs contain an entry with the same index and term, then the logs are identical in all entries up through that index. This is guaranteed by the leader consistency check in AppendEntries.
When a new leader is elected, it may have a log that is behind some followers. This is handled through a catch-up process: the new leader replicates its own log to followers, overwriting any uncommitted entries they may have from an old term. Only committed entries are preserved across leader changes.
This is the property that bites engineers when they look at follower logs during an incident. You might see a follower that appears to have more recent entries than the leader. Those entries are uncommitted artifacts from a previous election. They will be overwritten. Do not try to recover from them.
Raft in Production Systems
Understanding the algorithm is useful. Understanding how it manifests in the systems you operate is essential.
etcd: The Kubernetes Control Plane Foundation
etcd is built entirely on Raft and serves as the backing store for all Kubernetes cluster state. Every Kubernetes object, every deployment spec, every pod status update flows through etcd’s Raft log.
The default etcd configuration uses a 1000ms election timeout and 100ms heartbeat interval. This means it can take up to 1000ms to detect a failed leader and begin an election. In practice, combined with election and vote exchange time, a leader failure can cause a Kubernetes API server to be unresponsive for 2-5 seconds. If you have workloads that are sensitive to control plane availability, this matters.
Production etcd clusters should run with an odd number of nodes: 3 or 5 for most environments. Running 2 or 4 nodes provides no additional fault tolerance over 1 or 3 nodes because quorum requires a strict majority. A 4-node cluster can only tolerate 1 node failure (requires 3 of 4). A 3-node cluster also tolerates 1 failure (requires 2 of 3). The 4th node costs you an extra node for zero additional resilience.
The PostgreSQL HA setup with Patroni uses etcd as its distributed lock and leader election mechanism, which means etcd failures cascade directly to Postgres availability. I have seen clusters where engineers operated etcd as an afterthought and then wondered why Postgres was flapping during AZ instability.
CockroachDB: Multi-Region Raft
CockroachDB is built on top of a range-partitioned Raft implementation. Data is divided into ranges of roughly 64MB each, and each range has its own Raft group with its own leader. This is fundamentally different from etcd’s single Raft group covering all data.
This design means CockroachDB’s consensus topology is far more complex. A 9-node cluster might have thousands of Raft groups, each independently electing leaders and replicating logs. The benefit is that write throughput scales horizontally. The challenge is that it is much harder to reason about where leaders are located and what the write latency will be for any given row.
CockroachDB supports configuring the number of replicas per range (the default is 3) and where those replicas are placed using zone configurations. When running in a multi-region setup, you need to understand that a write to a range whose replicas span three regions will pay the full cross-region round-trip latency for every write to that range.
The distributed SQL comparison guide covers the operational trade-offs in more depth, but the Raft underpinning is why CockroachDB can claim ACID compliance across nodes.
Kafka KRaft: Raft for Metadata
Kafka’s migration from ZooKeeper to KRaft replaced an external coordination service with an internal Raft-based metadata log. Instead of all brokers connecting to a ZooKeeper ensemble for cluster metadata, the metadata is stored in a dedicated Raft log managed by a subset of broker nodes called controllers.
The Kafka KRaft migration guide covers the operational migration process. The architectural significance is that KRaft eliminated the ZooKeeper dependency that had historically been one of Kafka’s most operationally complex components.
The KRaft controller quorum is a typical Raft implementation. Controller nodes form a Raft group, elect a leader among themselves, and replicate metadata (topic configurations, partition assignments, leader epoch changes) through the log. The active controller corresponds to the Raft leader. Other nodes are followers.
A critical operational difference from etcd: Kafka’s data partitions are NOT managed by Raft. Raft only manages metadata. The actual message data in partitions is replicated through Kafka’s own in-sync replica (ISR) protocol, which is a separate mechanism that provides different consistency guarantees. This is a common source of confusion.
Consul: Service Mesh Coordination
Consul uses Raft for its catalog, KV store, and service mesh configuration. A Consul server cluster operates as a single Raft group, similar to etcd. Three to five server nodes is the standard configuration.
One underappreciated aspect of Consul’s Raft implementation is the stale read option. By default, Consul reads go through the leader to get linearizable consistency. But Consul also supports stale reads that can be served by any follower. Stale reads have no consistency guarantee but add zero write latency to the leader’s path and can handle load that would otherwise hit the leader.
This is actually a useful pattern to understand in general: Raft-based systems can often offer a consistency knob that trades read freshness for throughput and latency. etcd has a similar mechanism with its linearizable vs serializable read options.
Operational Implications for Architects
Knowing the algorithm helps, but here is what you actually need to internalize before you run Raft-based systems in production.
The Odd Number Rule
Always run an odd number of Raft nodes for your consensus quorum. I have seen countless engineers run 2-node etcd clusters because they thought it was more resource-efficient than 3. A 2-node cluster cannot tolerate any node failures. It requires both nodes to form a quorum. The moment one node fails, the entire system stops accepting writes. You have worse availability than a single node.
The minimum viable fault-tolerant configuration is 3 nodes (tolerates 1 failure). Five nodes tolerate 2 failures but require twice as many nodes to commit writes. For most production Kubernetes clusters, 3 etcd nodes is the right choice unless you have a strong reason for 5.
Election Timeout Tuning
The election timeout determines how quickly your system detects a failed leader. A shorter timeout means faster failure detection but more false elections from network blips. A longer timeout means more resilience to transient network issues but slower recovery from real failures.
In a well-tuned production environment with stable networking:
- Heartbeat interval: 100-200ms
- Election timeout: 1000-2000ms (5-10x heartbeat interval)
In high-latency environments (across AZs or regions):
- Heartbeat interval: 250-500ms
- Election timeout: 2500-5000ms
Never run your heartbeat interval close to your election timeout. You need enough headroom that a single delayed heartbeat does not trigger an unnecessary election.
Network Partition Behavior
When a network partition occurs, the behavior depends on which side has the quorum:
Majority partition (3 of 5 nodes): this side elects a leader if it does not already have one. It continues accepting writes. It is the “correct” side from Raft’s perspective.
Minority partition (2 of 5 nodes): this side stops accepting writes. Any existing leader on this side steps down when it cannot contact a majority. Reads may still be served from followers, but they may be stale.
Even split (in a 4-node cluster): this is why you do not run 4-node clusters. Neither side has a majority. Both sides stop accepting writes. The system is effectively down until the partition heals.
Understanding this behavior is critical for incident response. When you see etcd refusing writes, your first question should be: how many nodes are reachable from the leader? If the answer is less than a quorum, you need to focus on the network problem, not the application problem.
The Write Amplification Reality
Every write to a Raft-based system results in at minimum N network round trips (one to each follower) and N+1 fsync operations (leader plus each follower that acknowledges). For a 3-node cluster, a write that your application sees as a single operation is actually:
- Leader writes to its WAL (fsync)
- Leader sends AppendEntries to follower 1 and follower 2 (parallel)
- Followers write to their WALs (fsync each)
- Followers send acknowledgment to leader
- Leader commits and responds to client
This is why disk I/O performance matters enormously for Raft-based systems. etcd documentation recommends SSDs specifically because of this write pattern. Running etcd on spinning disks is a common and painful mistake. The fsync latency on a spinning disk can easily be 10-20ms, which directly adds to your write latency for every Kubernetes API operation.
The database reliability engineering principles apply here: measure your p99 write latency, not just average. Raft write tail latency is the latency of the slowest responding quorum member for each write, which means disk I/O outliers directly affect your client-visible latency.
Anti-Patterns I Have Seen at Scale
Twenty years of distributed systems work means I have watched engineers make the same mistakes repeatedly. Here are the ones that specifically relate to Raft-based systems.
Running Raft across regions without understanding latency impact. I worked with a team that deployed a 5-node etcd cluster spread across three AWS regions for “maximum resilience.” Kubernetes API calls were taking 200ms because every write needed quorum from nodes 150ms round-trip away. The cure was worse than the disease. For most Kubernetes clusters, keep etcd in the same region, across multiple AZs.
Treating Raft leader as a fixed node. I have seen load balancers configured to send all traffic to a specific node because “that’s the leader.” The leader changes. Configure your clients to discover the current leader, or better, send requests to any node and let the consensus library redirect non-leaders to the current leader.
Using Raft-backed storage for high-volume data. etcd has a maximum recommended database size of 8GB. It is not designed to store large amounts of application data, only coordination metadata. I have seen teams store large secrets, Helm release histories, and even application state in Kubernetes ConfigMaps, bloating etcd and causing performance degradation. The Kubernetes multi-cluster management challenges become significantly worse when your etcd is overloaded.
Not running etcd defragmentation. etcd uses a MVCC (multi-version concurrency control) model and accumulates old versions of objects. Without periodic compaction and defragmentation, the on-disk size grows unboundedly. Running etcdctl defrag and configuring automatic compaction is operational hygiene, not optional.
Ignoring clock skew. Raft does not use wall clock time for correctness, but many monitoring systems do, and extreme clock skew can cause issues with TLS certificate validation and metrics correlation during incidents. Run NTP/Chrony on all nodes and monitor clock skew as a production metric.
Geographic Distribution and Consensus
Multi-region Raft is a specific challenge that deserves its own discussion. The multi-region active-active architecture article covers the broader patterns, but the Raft-specific concern is write latency.
In a 3-node Raft cluster across three regions with 100ms round-trip between regions, every write waits for the first follower to acknowledge. In the best case (closest follower), this is 100ms of irreducible latency per write. For a coordination system like etcd, this is usually acceptable. For a transactional database, it often is not.
CockroachDB addresses this with follower reads: reads that are willing to read slightly stale data can be served from a local replica without consensus. This is similar to how read replicas work in managed PostgreSQL but with explicit consistency level controls.
Google Spanner takes a different approach: it uses atomic clocks (TrueTime) to bound clock uncertainty and can commit transactions with the certainty that they are globally consistent. This allows multi-region writes at Raft speed without the stale read trade-off, but the hardware requirement is significant.
For most teams, the practical answer to multi-region Raft is to keep your consensus quorum in a single region (or close AZs) and accept that your coordination service has a regional scope. Replicate data to other regions through asynchronous means and handle the consistency implications at the application level.
What This Means for Your Architecture
If you are running Kubernetes, you are running Raft. If you run CockroachDB, TiDB, or distributed SQL, you are running Raft. If you run Kafka 4.x, you are running Raft. If you use Consul for service mesh or configuration, you are running Raft. Understanding the algorithm is not academic curiosity: it is the operational prerequisite for reasoning about the failure modes of your most critical infrastructure.
The key mental model: Raft trades write latency and quorum complexity for the ability to survive node failures without manual intervention or split-brain. The trade-off is worth it for coordination services. It may or may not be worth it for your data layer, depending on your consistency requirements and throughput patterns.
The cluster sizing rules fall out of the algorithm itself: odd numbers, minimize cross-quorum-member latency, never exceed your capacity to lose nodes while maintaining a majority. The operational rules also fall out of the write path: SSD storage, tuned timeouts, regular compaction.
The GitOps tooling that depends on Kubernetes is only as reliable as etcd beneath it. The Kubernetes operators that manage your stateful applications rely on etcd for their coordination. Every layer of your cloud architecture that depends on Kubernetes state depends, ultimately, on five lines in the Raft paper about what it means for an entry to be committed.
Know the algorithm. Tune your clusters appropriately. Monitor your quorum health. And never, ever run a 2-node etcd cluster in production.
Get Cloud Architecture Insights
Practical deep dives on infrastructure, security, and scaling. No spam, no fluff.
By subscribing, you agree to receive emails. Unsubscribe anytime.