A FIELD GUIDE TO FAILURE
How complex
systems fail
A visual companion to Richard I. Cook’s essay on the hidden work that keeps things running
Start exploring
01 / GET THE FEEL OF IT
A small failure.
A whole system of defenses
Try this fictional service under a traffic surge. Change its defenses and follow the path
The load limiter catches the surge
The capacity buffer is unavailable. In this model, another layer still prevents the outage
A teaching model, with no probability estimates. Real defenses overlap, can share dependencies, and can fail together
MAKE A DEFENSE UNAVAILABLE
Look for combinations Cook treats catastrophe as a conjunction of failures Principle 03 ↗
02 / WALK THROUGH AN EXAMPLE
The incident starts
before the incident
A fictional checkout outage, viewed at three different distances
A new client starts retrying payments immediately after a timeout
A capacity upgrade is postponed while the team handles another launch
A promotion brings more shoppers. The dashboard still shows the previous five-minute average
The on-call engineer adds workers. Each new worker opens more database connections
Checkout stops responding
ORIGINAL FICTIONAL EXAMPLE · NO REAL INCIDENT IMPLIED
03 / THE COMPLETE MAP
18 ideas.
Four ways to see
A compact paraphrase of every principle, in its original order. Open any row for a fictional example and a question to bring to your own work
Showing all 18 principles
The four groupings are a reading aid created for this guide. Cook’s essay presents a single numbered sequence
04 / TRY A DIFFERENT PERSPECTIVE
You have
the outcome
The person making the decision had a different view. Step into a fictional on-call shift
Explore principle 08 ↗09:42 / WHAT YOU KNOW RIGHT NOW
- Checkout is slow for some customers
- The dashboard shows healthy database capacity
- Adding workers solved the last two slowdowns
- A promotion is live and support requests are rising
What would you try first?
NOW REVEAL ONE POSSIBLE OUTCOME
ORIGINAL THOUGHT EXPERIMENT · OUTCOMES ARE ILLUSTRATIVE
05 / TAKE IT WITH YOU
Make the next conversation
more useful
Original prompts for a team discussion, design review, or quiet moment of reflection
Start with these questions
- 01
What did the person know when they acted?
- 02
Which conditions had accumulated around that decision?
- 03
What helped the team notice, contain, or recover?
- 04
What would give the next person more time and better options?
A few useful terms
- Latent
- Present, with consequences that may emerge later
- Coupling
- How strongly a change in one part affects another
- Sharp end
- The point where someone must act on the real situation
- Emergent
- Arising through interactions among parts
- Hindsight bias
- Judging an earlier decision with knowledge acquired afterward
GO BACK WITH A DIFFERENT PAIR OF EYES
Now read Cook
Follow the original argument, with all the nuance intact
Read the original essay