Message → Distributed Consensus

Message → Distributed Consensus shows why a client timeout can follow a successful write. In a three-server Raft-backed service, two durable copies let the leader commit and apply a current-term command while the third server lags. If the response path fails afterward, the client cannot confirm that success. Byte delivery, storage, commitment, application, and receiving the result are separate events.

Think of the request as paperwork moving between three offices. A delivery receipt tells you the envelope arrived. A filed copy tells you one office recorded it. Approval under the offices' agreement rule is another event, and the reply informing you of that approval can get lost. Each receipt answers a narrower question than “is my work done?”

Two stored copies become evidence only when the reply arrives

The service stores a small variable, x. Client C sends request m42, asking it to run set x=7. Server S1 is already the elected leader of a three-server Raft cluster. S2 and S3 are followers. All three begin with the same log prefix through index 2, have applied that prefix, and hold x=0.

The request travels over an established TCP connection. TCP acknowledges received bytes, but its acknowledgment does not report whether the application parsed the command or wrote anything to disk. At the TCP ACK stop, the service's logs still end at index 2. TCP's reliable byte stream is the transport underneath the application protocol. RFC 9293

S1 accepts the command and stores a new entry at index 3 in its current term, 7. The notation 3/t7 names the entry's position and election epoch; it does not mean “third server” or “seven seconds.” S1 sends that entry to S2 in an AppendEntries RPC, which also names their matching prefix at 2/t6. S2 stores the entry durably before replying.

Storage and knowledge now differ. S1 and S2 both have the entry, but S1's replication evidence still lists only itself. The return RPC must arrive before S1 can count S2. For this current-term entry, two known durable copies meet the cluster's majority threshold and allow S1 to advance commitIndex to 3. It then applies the command in log order, advances lastApplied to 3, and changes x to 7. Persistent log state, commitment, and application are separate parts of the Raft write path. Raft paper, Figure 2 and §5.3

This service sends success after the leader applies the entry. S2 can still have commitIndex=2 at that instant, even though it stores 3/t7. A later heartbeat carries the new commit boundary; S2 then commits and applies locally. The service does not wait for every follower to apply before producing the response. Raft Apply lifecycle

Message to distributed consensus Step by Step

The default is Reply lost, with value 7. S3 is delayed throughout the run. Follow the receipts and the two boundaries on each server:

1. Deliver and acknowledge the bytes. C sends m42. S1's TCP stack receives the bytes and returns a TCP ACK. C records transport receipt, while every durable log still ends at 2.
2. Accept and store the command locally. S1 parses set x=7, then persists 3/t7. Its known durable copy count becomes 1/3; commitIndex=2, lastApplied=2, and x=0 remain unchanged.
3. Replicate to S2. AppendEntries carries 3/t7, the matching prefix 2/t6, and the old commit boundary 2. S2 persists the entry. Two disks now hold it, but S1 still knows about only one.
4. Receive majority evidence. S2's AppendEntries OK arrives. S1 knows about {S1, S2}, meeting the two-server threshold. The new entry's term matches the leader's current term. S3 has not participated.
5. Commit, then apply. S1 first moves commitIndex to 3 while its application remains at x=0. In the next beat, it moves lastApplied to 3 and sets x=7.
6. Lose the application response. S1 produces success for m42, but the client connection becomes unavailable before delivery and stays unavailable through the client's deadline. C times out with an unknown outcome. S1 retains its committed entry and x=7.
7. Spread the commit boundary. Independently of the client deadline, S1 sends a heartbeat with leaderCommit=3. S2 advances commitIndex, then lastApplied, and sets x=7. S3 still holds the older prefix and x=0.
8. Compare fact with evidence. The final cards put the uncertain client beside the committed and applied service result. C did not receive the application result that would let it confirm success.

The seven-chip artifact chain records what each stage produced. A stored entry supplies replication; majority evidence enables commitment; a committed entry supplies application. The final chip requires a result that actually reaches the client. Earlier successful stages remain present when that last handoff breaks.

Why a client timeout leaves the outcome uncertain

Change only the scenario and keep value 7. The three runs make different failures comparable:

In No quorum, S2's replication path is also unavailable. S1 stores the command but cannot collect a second durable copy. The run ends with an uncommitted entry. That is what the observer can see; the client has only its timeout. The client sees the same missing response in Reply lost, where the command has already executed.

A timeout therefore belongs to the caller's waiting policy. Treating it as a rollback instruction would confuse an observation at the client with a mutation on the server. A pending entry may later commit or be replaced during recovery; this animation does not choose its future.

Retries need their own application contract. This example uses set x=7, but other commands might increment a counter or charge an account. A correlation ID alone does not provide deduplication. Raft orders log entries; a service that must suppress repeated client commands needs additional request identity and result-tracking machinery. The paper discusses that distinction in its client interaction section. Raft paper, §8

For example, replace this command with increment x. If the first attempt runs but its response is lost, submitting a second log entry can increment twice. In code that calls such a service, keep the timeout outcome uncertain and follow the API's documented status-check or idempotent-retry contract. Consensus alone does not make a new submission the same operation as the old one.

Edge Cases

• Writing the existing value. Set the input to 0. Applying set x=0 leaves the visible value unchanged, but lastApplied still advances to 3. The command was processed even though its value matches the old one.
• One delayed follower. S3 remains at index 2 in all scenarios. Majority progress does not require simultaneous application on all machines. A lagging prefix is different from applying a conflicting command at the same index.
• Two stored copies before the leader knows. Pause on S2's returning acknowledgment: two logs contain 3/t7, while S1's evidence count is still one. The leader cannot act on a reply that has not arrived.
• Current-term restriction. This trace's direct majority commitment concerns an entry created in the current leader's term. Older-term entries require Raft's additional commitment reasoning; copying the visible counting rule without that condition is incomplete.
• Response-path failure. The model holds the client connection unavailable through its deadline. One lost TCP segment would ordinarily be retransmitted; packet loss alone is not presented as permanent loss of a reliable stream.
• Signed values. The input accepts integers from -99 through 99 for legible labels. That is a visualization bound, not a Raft limitation.

Common Mistakes

• Calling every acknowledgment the same thing. TCP ACK confirms byte receipt. AppendEntries OK confirms a follower's storage of the matching log entry. The application response confirms the service result to the client.
• Counting the client as a replica. The majority is among S1, S2, and S3. C's transport acknowledgment supplies no additional log copy.
• Reporting success after one local append. No quorum preserves that very state: real disk storage without majority commitment.
• Applying an entry because it is stored. S2 stores 3/t7 before it learns that index 3 is committed. Its application waits for that boundary.
• Turning every server green at once. Each server learns and applies locally. The later commit notification has information S2 did not receive in the first replication RPC.
• Interpreting timeout as “did not execute.” Reply lost ends with lastApplied=3 on S1 and an uncertain client. A missing reply cannot undo that execution.

A Note on Simplification

This is an explainer, not a substitute for formal study of distributed systems. It fixes three non-Byzantine servers, a previously elected leader, matching initial logs, and one current-term write. Connection setup and TCP recovery are grouped; elections, routine heartbeats, persistence latency, batching, message reordering, leader replacement, older-term commitment, reads, client retries and deduplication, snapshots, and membership changes are omitted. Messages are staged one at a time, and playback seconds are not network measurements. The response policy waits for leader application; other service APIs may expose different completion contracts. Continue with Raft Consensus for election and log-repair reasoning, or TCP Handshake for the connection underneath this request.