Rule 29 of 33 · Chapter VI — When Things Break
Fix it so it can't happen the same way twice
Why this rule exists
An incident you resolve but don't learn from is an incident you've agreed to have again, because the conditions that produced it are still sitting there waiting. The real payoff of an outage isn't the heroics of the fix, it's the durable improvement that makes that entire class of failure impossible or at least loud and early next time. Every incident is expensive tuition, and we only get our money's worth if we turn the specific failure into a systemic fix: the guardrail, the test, the alert, the automated check that would have caught it. Without this discipline, teams live in a cycle of firefighting the same fires, feeling busy and heroic while never actually getting more reliable. The goal is a system that gets sturdier every time it breaks, where each incident permanently removes a way things can go wrong. Reliability isn't the absence of failure, it's what you build out of failures you refused to waste.
In practice
Every postmortem ends with concrete, owned action items aimed at prevention, not just 'be more careful,' which is not a fix. Prefer systemic guards, an automated check, a test, an alert, a safer default, over relying on humans to remember, because humans will forget and the guard won't. Actually do the action items, treating them as real work with owners and dates, not a wishlist that decays after the adrenaline fades. Address the class of problem, not just the exact instance, so you close the door on the whole category rather than one specific way through it. When an incident reveals that recovery was slow, fix the recovery too, faster rollback and better alerting are prevention for next time. And track whether the same kind of thing recurs, because a repeat is a sign the last fix addressed the symptom, not the cause.
When it doesn't apply
Not every action item is worth doing; some prevention costs more than the failure it prevents, so weigh the fix against the likelihood and severity honestly rather than gold-plating against a freak event. And in the immediate aftermath, stabilize first, deep systemic fixes come after the fire's out, not during it.