network partitions
what actually happens when nodes cannot reach each other, not just the theory.
a network partition is a group of nodes that cannot exchange messages with another group. not slow, not lossy. no communication at all. the cluster has become two clusters that each think they might be the real one.
they are also not exotic. misconfigured switches, BGP changes, a firewall rule rolled out to the wrong subnet, a cloud provider replacing hardware, an NIC that fails in one direction only. bailis and kingsbury's the network is reliable collects the public postmortems, and the jepsen analyses are essentially a decade-long record of databases finding out what they actually do under partition, which is frequently not what their documentation claims.
the view from inside
three nodes: A, B, C. the link between A and the {B, C} pair fails. B and C can still reach each other.
A sends to B and C and hears nothing back. from A's position, these are all consistent with the evidence:
- B and C have crashed
- B and C are healthy and the network between them and A is broken
- the failure is one-directional. A's messages are arriving, and the replies are not coming back
that third one is the nasty case. A's writes may be landing on B and C while A concludes they are unreachable.
from B and C's side, A has gone quiet. A might be dead. A might be happily accepting client writes from a load balancer that can still reach it.
this is the defining property: an isolated node cannot distinguish a partition from a crash. the symptoms are identical. the consequences are not. a crashed node comes back with nothing to reconcile, while a partitioned node comes back with a divergent history.
the decision
a node that cannot reach a quorum has to do something, and there are only three somethings.
keep serving. accept reads and writes without coordination. reads may be stale, writes may conflict with the other side, and healing requires reconciliation. this is the AP choice.
stop serving. refuse anything that cannot be confirmed by a quorum. reads error, writes are rejected, the minority side is dark until the network returns. nothing diverges. this is the CP choice.
serve reads, refuse writes. read-only mode. stale reads are acknowledged as such; nothing new can conflict. it is a genuinely useful middle ground and it is underused, because it preserves the read path, usually most of your traffic, while making divergence impossible.
which is correct depends entirely on the data. a session cache serving a stale preference is fine. a ledger accepting a debit that the other side will never see is not.
split brain
the failure this all exists to prevent has a name. split brain is both sides deciding they are authoritative and both accepting writes.
quorum is the standard defence: require a majority to act, and since two disjoint majorities cannot exist in one cluster, at most one side can proceed. this is why cluster sizes are odd. a 3-node cluster survives losing 1; a 4-node cluster also only survives losing 1, so the fourth node buys nothing and adds a coordination partner.
quorum is necessary and not sufficient. a node can be slow rather than partitioned, lose its lease, and resume acting on a stale belief that it still holds it. this is why lease-based systems use fencing tokens: a monotonically increasing number issued with the lease, which downstream storage checks and rejects if it has already seen a higher one. without fencing, a garbage-collection pause long enough to expire a lock is sufficient to corrupt data, and GC pauses of that length are not rare.
healing
when the network comes back, the work depends on what happened during the split.
if both sides refused writes, there is nothing to merge. the lagging side catches up through normal replication and the incident is over.
if both sides accepted writes, you need conflict resolution:
last-write-wins compares timestamps. it is simple, fast, and silently lossy. clocks across machines disagree by more than the interval between conflicting writes, so "later" is often wrong and the discarded write is gone with no record.
vector clocks track causality explicitly, so the system can tell a genuine concurrent conflict from a stale overwrite and surface only the real conflicts. dynamo is the canonical description. the cost is metadata that grows with the number of writers, and an application that must know what to do with a conflict.
CRDTs are data types designed so concurrent operations always merge without a conflict ever arising. excellent when your data fits one of them: counters, sets, some maps, collaborative text. not a general answer, because most business logic does not have the algebraic properties that make it work.
whichever your database uses, it has already chosen for you. that is worth knowing before an incident rather than during one, because the failure mode is invisible: lost writes leave no error in any log.
partial partitions are worse
a clean split is the tractable case. real networks often deliver something messier. 80% of packets get through, or A reaches B but B cannot reach A, or the link works for small packets and fails for large ones after an MTU change.
these break the assumptions underneath quorum. a node that is intermittently reachable can flap in and out of the cluster, triggering repeated leader elections, each of which stalls the cluster while it completes. the cluster spends its time deciding who is in charge instead of doing work.
most designs reason about clean partitions because clean partitions are analyzable. it is worth being explicit that this is a simplifying assumption and that reality is harder than the model.
defining the behavior is the point
your system will be partitioned. the only question is whether its behavior during that partition is something you chose.
a system with no explicit partition handling still behaves somehow, usually every node operating independently, state diverging, and an undefined mess to untangle afterwards. that is a design decision made by omission.
"go read-only when we lose quorum" is a modest-sounding policy and it is worth more than any amount of hoping the network holds.