Designing for Failure Domains, Not Just Failures

A few years ago, a single slow dependency took down an entire client platform for eleven minutes. The dependency itself recovered in four seconds. The other ten minutes and fifty-six seconds were us — specifically, a retry policy that turned one slow service into a self-inflicted denial-of-service against our own database.

That incident changed how we think about system design. We stopped asking "how do we handle this failure?" and started asking "how far should this failure be allowed to travel before it's contained?"

What a failure domain actually is

A failure domain is the blast radius of a single failure — the set of things that go down together when one thing goes down. Most teams draw failure domains around services. We've found it's more useful to draw them around shared resources: connection pools, thread pools, caches, and queues that multiple otherwise-unrelated code paths depend on.

The retry storm, in detail

Here's roughly what happened. A downstream pricing service slowed down. Every caller retried on timeout — reasonable in isolation. But every retry held a connection from a shared pool a few seconds longer, and the pool has a fixed size. Within about ninety seconds, the pool was fully exhausted by requests waiting on a service that had, by then, already recovered. Unrelated services sharing that pool started failing too, for a dependency they'd never even called.

// The naive retry that caused it
async function callPricingService(req: Request) {
  for (let attempt = 0; attempt < 5; attempt++) {
    try {
      return await pricingClient.call(req); // holds a pooled connection the whole time
    } catch {
      await sleep(200 * attempt); // no ceiling, no circuit breaker
    }
  }
  throw new Error("pricing service unavailable");
}

What we changed

Three changes, none of them exotic, all of them now mandatory in our architecture reviews:

• Bulkheads around shared resources. Every external dependency gets its own connection pool, sized deliberately small, so exhaustion in one pool can't starve unrelated callers.
• Circuit breakers with a real open state. After a threshold of failures, the breaker opens and fails fast for a cooldown window — no more retries hammering a struggling dependency back toward the service that's waiting on it.
• Retry budgets, not retry counts. Retries are capped as a percentage of total request volume per time window, not a fixed per-request count, so a system-wide slowdown can't multiply itself.

Failure domain map showing shared connection pools across services

Drawing the failure domain map

We now maintain an explicit diagram for every production system: not the service architecture diagram everyone already has, but a failure-domain map — which shared resources exist, which services draw from them, and what happens to unrelated callers if that resource saturates.

“"The architecture diagram tells you how the system is supposed to work. The failure-domain map tells you how it's actually going to fail."

Building that map for our platform surfaced four other shared-pool risks we hadn't noticed, all fixed before they became incidents. That's the actual return on this exercise — not the postmortem fix, but the map that finds the next one before it happens.