latency vs outage
they look different, are detected differently, and need different responses.
when a dependency starts misbehaving, the first thing worth establishing is which kind of misbehaving. down and slow produce different signals, arrive on different timescales, and call for opposite responses. treating one as the other is how incidents get longer.
two shapes
an outage means connections are refused or requests error immediately. connection refused, 500 in 2ms, TCP reset. the request fails and the caller's thread comes straight back.
latency degradation means requests are accepted and processed, slowly. nothing fails. the caller waits. the response arrives in 2 seconds instead of 20ms, or it does not arrive and eventually the timeout fires.
they look completely different on a dashboard, and the difference is not just magnitude:
the counterintuitive part is the latency panel during an outage. p99 latency improves when a service goes down, because ECONNREFUSED returns in microseconds. a latency graph alone will show a service that just died as having gotten faster.
the reverse gap is the dangerous one. during degradation the error rate stays flat until your timeout fires, which might be 30 seconds after latency started climbing. anything alerting purely on errors is blind through exactly the window in which
why the response differs
if a service is down, the answer is unambiguous: stop sending it traffic, fail fast, serve from cache or a degraded path. circuit breakers open, the load balancer pulls the instance, and the system routes around a thing that is definitively not working.
if a service is slow, the same actions can be actively harmful. a service overloaded by a traffic spike may be working through a backlog and recovering on its own; cutting its traffic to zero and then restoring it all at once produces a thundering herd against a cold cache. a service that is slow because it is waiting on something else does not benefit from you removing its instances.
so "down" has one correct response and "slow" has several, and choosing between them requires knowing the cause:
resource contention (CPU, memory, disk I/O) means the service needs less traffic. a slow dependency means the problem is one hop further down and the service in front of you is a victim. a load spike means back-pressure and shedding. a GC pause is often transient and needs nothing at all. lock contention on a specific query needs that query fixed and nothing else.
a 200ms GC pause does not warrant opening a circuit breaker. a blocking query that has stalled every connection does. the tooling cannot tell these apart for you.
detecting each
for outages, error rate is the signal. connection refused, resets, explicit 5xx. it moves immediately and unambiguously, and health checks fail at the same time.
for degradation, you need latency percentiles, and you need the right ones. the mean is close to useless. it is dragged around by outliers and it hides the shape entirely. p50, p95, p99 as separate series tell you whether everything got slower or a subset of requests fell off a cliff, and those have different causes. histograms beat both, because they let you ask questions after the fact that you did not know to ask in advance.
the tail matters more than it looks. dean and barroso's the tail at scale shows why: fan out one request to 100 servers with a 1% chance each of being slow, and about 63% of top-level requests hit at least one slow server. a p99 problem in a leaf service is a common-case problem at the edge.
a rough triage heuristic that holds up in practice:
errors high, latency normal → something is failing fast. outage-shaped. latency high, errors normal → something is slow. degradation-shaped, and you are early. both high → degradation that has been running long enough for timeouts to fire. you are late, and the cascade may already be moving.
timeouts convert one into the other
a timeout turns latency into errors by definition. set 5 seconds, and a request that would have hung for 60 now fails at 5. this is good. it is what bounds concurrency and prevents the cascade.
the side effect is that your monitoring changes shape at the timeout boundary. the service is slow, not down, but callers see errors climbing and every outage-response mechanism starts firing on a service that is still processing requests.
sometimes that is correct. a service answering in 30 seconds is functionally unavailable and treating it as down is honest. sometimes it is not: a service answering in 6 seconds behind a caller with a 5-second timeout is being declared dead by a threshold rather than by its own behavior. before concluding the dependency is broken, check whether the timeout is calibrated to its actual p99. a surprising share of "the service is failing" turns out to be "we picked 5 seconds because it sounded reasonable in 2019".
in an incident
the first diagnostic question is whether you are looking at fast failures or slow failures, because it determines everything after it.
fast failures point at crashes, restarts, deploys, connectivity, and the service's own error responses. slow failures point at resource contention, a slow dependency one hop down, load spikes, GC, and lock contention.
answering it takes one look at whether error rate or latency moved first. getting it wrong sends the team to the wrong layer, and in the slow case it also means the clock is running on a cascade nobody has noticed yet.