A leader is not an oracle. During a partition, a server can still believe it is the leader long after the rest of the cluster has elected someone else. The useful question is what evidence that server needs before it can acknowledge a write or answer a read.
This lab adapts the actual protocol rules from Diego Ongaro’s RaftScope. Requests and replies travel separately. A click does not quietly deliver a whole round of RPCs. You can delay an acknowledgement, crash its recipient, or deliver an older response after a newer one. That distinction is where an election animation starts becoming a useful debugging tool.
Follow one write all the way to its acknowledgement #
Start an election on S1, then advance the clock. RequestVote requests leave first; the votes arrive later. After S1 becomes leader, append an entry. The new cell is initially pending even though S1 has already written it locally.
Watch the leader’s per-peer matchIndex. It records the highest index the leader knows a follower has replicated. nextIndex is where it will try the next replication RPC. These are different kinds of state: one is evidence, the other is a retry cursor.
An AppendEntries request carries the preceding index and term. A follower accepts the suffix only when that prefix matches. A successful reply supplies replication evidence; a failed one makes the leader back up and retry. Delivering a request and losing its reply can therefore leave the follower ahead of what the leader knows. That is normal, not corruption.
The commit index advances once the leader has sufficient evidence. Followers learn that advancement in later AppendEntries messages, so their blue cells need not change simultaneously.
Break availability without breaking safety #
Load Divergent logs scenario. S1 and S2 have an uncommitted second entry. The other three replicas have elected S3 and committed a conflicting entry at the same index.
Both S1 and S3 can display “leader.” They are in different terms. Raft forbids two elected leaders in the same term; it does not require every isolated process to instantly learn about a newer one.
Try another append on S1. It can accept the request locally but cannot acknowledge a committed write with only two replicas. A client timeout at this point means the outcome is unknown to the client, not necessarily that no server stored the request.
Heal the network and advance 300 ms. The higher term makes S1 step down. The prefix checks let the new leader replace the conflicting, uncommitted suffix. The first committed entry survives.
In a service, this is also where request identity matters. A retry after an ambiguous timeout can execute twice unless the replicated state machine deduplicates requests. Raft orders commands; it does not automatically make an application operation exactly once.
The current-term rule is not a footnote #
A leader cannot commit an older-term entry merely because it now appears on a majority. Raft’s commit rule requires a current-term entry at the index being advanced:
The leader counts its own copy. Once that entry commits, the preceding prefix commits with it. The term restriction closes an election-and-overwrite case that “three copies means committed” misses.
The lab uses RaftScope’s rule rather than substituting a majority counter. For the full argument, read Figure 8 and the safety discussion in the Raft paper. A finite simulation is a way to inspect the argument, not a replacement for it.
A read needs a freshness argument #
Reading a replica’s locally committed prefix can return stale data. A minority leader may know about an earlier committed value while another leader has already completed a newer write.
Read through log barrier takes the deliberately expensive route: append a fresh current-term marker and wait for it to commit. A state machine that has applied through that marker can answer the read at that point in the history. This is a protocol illustration; the lab does not execute an application state machine.
Production implementations often avoid a log entry for every read. ReadIndex-style protocols establish current-term leadership, confirm it with a quorum and wait for the local applied index. Lease reads add timing and clock assumptions. Those are different contracts, not interchangeable “fast read” switches. The lab does not claim to implement either optimization.
Recovery is about durable state, not the color of a node #
Crash a server and restart it. Its term, vote and log survive. Its volatile commit index resets and is learned again from the leader. Forgetting a durable vote after a crash would permit a server to vote twice in one term; forgetting a acknowledged log entry would invalidate the replication evidence on which safety depended.
Here, persistence is instantaneous and reliable before messages are sent. There is no disk queue, torn write, snapshot installation, compaction, membership change or Byzantine behavior. The clock advances in 5 ms increments, using the source’s short illustrative timeout settings. These are not deployment recommendations.
For a real incident, I would want the same evidence exposed here—term transitions, election causes, peer progress and outstanding requests—alongside fsync latency, scheduler stalls, network RTT tails and applied-index lag. A healthy average RTT does not explain a cluster that repeatedly loses leadership during storage stalls.
Make the failure reproducible #
Back one action restores the random-number state as well as the replicas and in-flight messages. Repeating an action therefore repeats the timing choices. That makes it possible to change one delivery decision and inspect what actually depended on it.
The safety inspector checks observed leader uniqueness per term and agreement of committed entries across the run. “Holding so far” is intentionally weaker than “correct.” There is no model checking or exhaustive schedule exploration behind that label.
Protocol source: Diego Ongaro’s RaftScope, revision 5b0c10ab, under its ISC licence. The local additions are seeded replay, partitions, crash/restart semantics, a message inspector, read markers and explicit safety observations. The original source and the Raft paper remain the references for the algorithm.