reliability

a reliable system keeps doing what it is supposed to do, at the expected level of performance, while things go wrong underneath it. things go wrong constantly, and the useful mental shift is treating that as the normal operating condition rather than the exception.

the failure rate is not an abstraction

hardware failure is easy to hand-wave about, so it is worth using real numbers. backblaze publishes failure statistics for its entire fleet; the 2025 drive stats cover 344,196 drives with an annualized failure rate of 1.36%.

fleet: 344,196 drives, AFR 1.36%
expected failures/year : 4,681
                 /week : 90
                  /day : 12.8

thirteen drives a day, and not because it was a bad month. that is the steady state. at that fleet size, "a disk failed" is not an incident, it is tuesday, and any design that treats it as an incident will generate roughly 4,681 pages a year.

vendor numbers will not tell you this. schroeder and gibson's disk failures in the real world compared datasheet MTTF against field replacements across 100,000+ drives: datasheets claiming 1–1.5 million hours imply an annual failure rate under 1%, while observed replacement rates were commonly 2–4% and reached 13% on some systems. the datasheet describes a drive under ideal conditions. your fleet is not under ideal conditions.

the same shape applies to everything else in the rack. memory bit flips, silent network hardware degradation, a CPU with an erratum. at scale these stop being rare events and become a background rate you design against.

software and people

software fails differently. hardware failure rates are roughly stationary; software defects are correlated, because every replica runs the same build. a race condition that only appears above a certain load appears on all of your nodes at once, which is exactly the correlated failure that redundancy does not help with.

operator error deserves equal billing and usually gets less. misconfigured deploys, wrong firewall rules, a migration run against the wrong database. it is one of the most common causes of serious incidents, and the reason is structural: a human with sufficient permissions is a single point of failure that no amount of replication addresses. a system that cannot survive a wrong command is not reliable, however good its hardware story is.

faults are not failures

a fault is one component deviating from its spec. a failure is the system as a whole no longer providing its service. the entire discipline is about keeping the first from becoming the second.

redundancy is how you do that, and it does not reduce the fault rate at all. it contains faults so they stop short of failure. this is why fault tolerance and fault avoidance are different strategies. avoidance tries to stop faults happening: testing, review, better hardware. it has a hard ceiling, because the interesting faults are the ones nobody predicted. tolerance assumes faults happen and asks what the system does next. you need both, but only one of them scales.

mtbf, mttr, and which one to work on

mtbf (mean time between failures) is how long a component runs before failing. it is a distribution, not a promise: a 3-year MTBF means half the units fail sooner.

mttr (mean time to repair) is how long it takes to restore service afterwards.

availability comes out as MTBF / (MTBF + MTTR), so both terms matter, but they are not equally tractable. raising MTBF means better hardware, more testing, fewer changes, and it fights against shipping. lowering MTTR means faster detection, faster rollback, better runbooks, and it is mostly engineering you control directly.

a system that fails weekly and recovers in 10 seconds is more available than one that fails yearly and takes 6 hours. it also feels worse to operate, which is why teams often spend their effort on the term that is harder to move. if you are choosing where to put a quarter, MTTR usually pays better.

composition, again

reliability composes the same unforgiving way availability does: three dependencies at 99.9% each put you at roughly 99.7% before you have written any code. add a database, a cache, and a queue and it keeps falling. the boundary of your service is not the boundary of your reliability problem.

graceful degradation is the lever that changes the arithmetic. if your service can run with reduced functionality when a dependency is gone, serving search without personalization or a page without recommendations, then that dependency's outage stops being multiplied into your own number. degraded is a state you designed. down is one that happened to you.

reliability and availability pull apart

they are routinely conflated and they can move in opposite directions.

a batch job that runs hourly, takes 30 minutes, and has never produced an incorrect result is offline half the time. poor availability, excellent reliability. a cache that is always reachable and occasionally serves data from three hours ago is the reverse.

the distinction is not pedantic, because the fixes differ. correctness problems are addressed with validation, consistency models, and testing. reachability problems are addressed with replication, failover, and load balancing. optimizing one does nothing for the other, and knowing which one you actually have is most of the work.