Building Idempotent APIs at Scale
A retried payment request charged a customer twice. The fix wasn't better error handling — it was accepting that every write endpoint needs to answer "what if this exact request arrives again?"
The incident that changed how we build write endpoints was small in scope and large in consequence: a mobile client retried a payment request after a timeout, the original request had actually succeeded, and the customer was charged twice. The client did nothing wrong — a timeout genuinely doesn't tell you whether the server processed the request or not.
Why "just add a timeout retry" isn't safe by default
Retrying on timeout is standard, reasonable client behavior. The problem is entirely on the server side: most write endpoints assume each incoming request is new. Without an explicit mechanism to recognize "I've seen this exact request before," a retry of a successfully-processed request just processes it again.
Idempotency keys, done properly
The fix is a client-generated idempotency key attached to every write request, checked against a short-lived store before any side effect runs.
async function handlePayment(req: PaymentRequest) {
const existing = await idempotencyStore.get(req.idempotencyKey);
if (existing) {
return existing.response; // replay the original result, don't reprocess
}
const result = await chargeCard(req);
await idempotencyStore.set(req.idempotencyKey, result, { ttl: "24h" });
return result;
}
The idempotency check and the side effect have to be part of the same atomic operation, or you've just moved the race condition instead of closing it. We use a database transaction with a unique constraint on the key, not a separate cache lookup before the charge.
Where this gets harder than it looks
Idempotency keys are easy to describe and easy to get subtly wrong. Two failure modes we hit before getting this right:
- Storing the key before the operation completes, which means a crash mid-request leaves the key marked "seen" with no response to replay — the retry then hangs or errors instead of safely repeating.
- Scoping keys too broadly, where two genuinely different requests from the same client accidentally reuse a key and one silently returns the other's result.
Idempotency key check and side effect executed as one atomic operation
What we now require on every write endpoint
Every endpoint that causes a side effect — payments, provisioning, sending a notification — requires an idempotency key as part of its contract, not an optional header. Client libraries generate one automatically per logical operation, and the server-side check-and-store is atomic with the operation it's guarding.
"A write endpoint that isn't idempotent doesn't have a bug yet. It has a bug waiting for the first network blip, which on the internet is not a matter of if."
Since making this mandatory across our platform APIs, we've had zero duplicate-side-effect incidents from retries — and just as importantly, client teams stopped writing defensive de-duplication logic on their end, because the guarantee now lives where it actually belongs.
One email, every other Friday
Engineering notes, new articles, and things we learned the hard way — no marketing, ever.
Read by 2,400+ engineers. Unsubscribe anytime.
Let's talk
Let's build what's next.
If any of this sounds like the kind of work you want done on your systems, our engineers are the ones who'll actually do it.