Shipping Faster With Progressive Delivery
We stopped asking "is this safe to deploy" and started asking "how little blast radius can this deploy have." That shift cut our rollback time from 40 minutes to under 90 seconds.
For years, our deploy process had one gate: a staging environment that looked "close enough" to production. It caught obvious breakage and missed everything that only showed up under real traffic patterns — which, in practice, was most of what actually caused incidents.
The problem with a single gate
A single pre-production gate makes an implicit bet: that staging traffic is representative of production traffic. It rarely is. Our worst incidents in the old model were things staging couldn't have caught — cache stampedes under real concurrency, a query plan that only degraded past a certain data volume, a memory leak that took hours of sustained load to surface.
# Simplified shape of our current rollout stage
stages:
- canary: { traffic: 5%, duration: 10m, auto_rollback_on: [error_rate, p99_latency] }
- partial: { traffic: 25%, duration: 20m, auto_rollback_on: [error_rate, p99_latency] }
- full: { traffic: 100% }
What progressive delivery actually changed
Instead of one gate, every deploy now moves through real production traffic in increasing slices, with automatic rollback on error-rate or latency regression at each stage. A bad deploy today reaches five percent of production traffic for at most ten minutes before it's automatically rolled back — not forty minutes of a full incident before a human notices and rolls back manually.
Automatic rollback only works if your rollback is actually fast and actually safe. We didn't trust auto-rollback until we'd tested it under real failure conditions a dozen times — it has to be boring and reliable before it can be automatic.
Canary traffic ramp with automatic rollback thresholds
The part that mattered more than the tooling
The tooling was the easy part. The harder change was cultural: engineers had to trust that a five-percent canary would actually catch problems, and stop manually babysitting every deploy "just in case." That trust took a few real caught-in-canary incidents to build, but once it did, deploy frequency roughly tripled — because shipping stopped feeling risky.
One email, every other Friday
Engineering notes, new articles, and things we learned the hard way — no marketing, ever.
Read by 2,400+ engineers. Unsubscribe anytime.
Let's talk
Let's build what's next.
If any of this sounds like the kind of work you want done on your systems, our engineers are the ones who'll actually do it.