Skip to main content

How I Write Postmortems

A useful postmortem should make the same failure harder to repeat, easier to detect, or cheaper to recover from.

Lovell Felix 6 min read

A postmortem can document an incident perfectly and still fail to prevent the next one.

I have seen write-ups with complete timelines, screenshots, chat transcripts, and long lists of follow-up items that still left the important question unanswered: what about the system needs to change?

That is the part I care about most when I write one. The record matters, but the useful output is whether the same failure becomes harder to repeat, easier to detect, or cheaper to recover from.

Over time, that has made me more selective about what belongs in the document. I want enough detail to reconstruct the failure, explain how it reached users, and show what changed in the system afterward. I do not want completeness for its own sake.

The timeline should explain the incident

I always want a timeline, but I do not treat it as a transcript.

During an incident, every command, alert, message, and theory can matter. In the postmortem, most of that activity becomes noise unless it explains a turning point in the response.

The events I usually care about are the ones that show when the system first became unhealthy, when we detected it, when the working theory changed, when mitigation reduced impact, and when recovery was actually validated.

That distinction matters because mitigation and recovery are not always the same thing. Traffic can stabilize while a dependency is still unhealthy. A rollback can finish while stale state remains somewhere else in the system. An alert can clear before the condition that caused the incident is fully understood.

A useful timeline makes those boundaries visible. It should help someone reading the document months later understand where detection was slow, where diagnosis went in the wrong direction, and what evidence finally showed that the system had recovered.

Timestamps that only prove activity rarely help with that.

I separate the trigger from the failure path

I am suspicious of postmortems that arrive at one clean root cause too quickly.

Sometimes there is an obvious trigger: a bad deployment, an expired certificate, a dependency outage, a configuration change, or a broken flag. But the trigger alone usually does not explain why the incident had the impact it did.

A deployment can be wrong without becoming a large incident. The more useful questions are what allowed it through, why the rollout continued, why the signal that should have stopped it was weak or missing, how wide the blast radius became, and what made recovery take as long as it did.

That is the failure path I want the postmortem to reconstruct.

For example, ending the analysis at “bad deploy” usually produces a weak corrective action such as “be more careful during deployment.” If the actual path was that validation missed the condition, the rollout window was too aggressive, the metric used for gating was noisy, and rollback depended on a manual step, there are several concrete places to improve the system.

The same is true outside deployment failures. A dependency outage becomes more interesting when the postmortem explains why clients kept retrying, why isolation was insufficient, or why the degraded path was not usable when it was needed.

The trigger tells me what started the incident. The failure path tells me why it became one.

Blameless does not mean vague

I do not find postmortems useful when they turn a human decision into the root cause.

People make decisions with the tooling, documentation, signals, permissions, and time available to them. If someone skipped a check because the runbook was ambiguous, approved a rollout because the available telemetry looked healthy, or chose a mitigation that made sense with incomplete information, I want the write-up to say what happened clearly.

The important part is what made that decision reasonable at the time and what prevented the system from catching it earlier.

That is where useful follow-up work usually appears. Maybe the dangerous operation should not have been possible without an additional guardrail. Maybe the approval path had no visibility into the metric that actually represented risk. Maybe recovery depended too heavily on someone remembering a rarely used procedure while the system was already failing.

Blame tends to shorten the analysis at exactly the point where it becomes useful.

Action items should change the operating conditions

Generic follow-up items are one of the fastest ways for a postmortem to turn into paperwork.

“Improve monitoring,” “update documentation,” and “review the deployment process” all sound reasonable. None of them say what will be different the next time the same condition appears.

I prefer action items that change an operating condition.

That might mean adding a deployment gate tied to a specific failure signal, changing the rollout order so the blast radius stays bounded, removing a manual recovery step, or tightening an alert so it maps to a decision somebody can actually make.

Documentation can be the right fix too, but I want to know what uncertainty it removes. “Update the runbook” is weak. “Rewrite the rollback procedure so the preconditions, command, and validation steps are clear without following three separate links” is something I can evaluate later.

I also try not to turn every weakness exposed by an incident into a project. A failure can reveal ten things that could be better without making all ten equally important. If two changes remove most of the recurrence risk, I would rather make those changes explicit than create a long backlog that slowly loses connection to the incident.

Sometimes the right follow-up is removal

Incidents also expose operational machinery that no longer earns its keep.

A dashboard may still exist because it was useful for an older architecture. A runbook may contain steps that no longer match the current deployment path. An alert may fire reliably but provide no information that changes the response. A mitigation may remain documented long after the system changed underneath it.

Those things create cognitive load at the worst possible time.

I want a postmortem to make it acceptable to remove them. More monitoring, more documentation, and more process do not automatically create a safer system. Sometimes the safer system is the one with one fewer ambiguous step, one fewer noisy alert, or one fewer path responders have to evaluate while production is already degraded.

This is one reason I try to keep the follow-up list small. The goal is not to prove that the incident produced enough work. It is to change the parts of the system that made the incident possible or expensive.

The review should challenge the explanation

By the time a postmortem reaches review, the draft can feel settled simply because somebody spent hours reconstructing the incident.

That is when I want people to challenge the explanation.

Did we stop at the trigger instead of following the path to impact? Did we call something a communication failure because that was easier to see than the missing system signal underneath it? Are we proposing process because the technical guardrail is harder to build? Did mitigation work for the reason we think it did? What evidence actually established recovery?

A useful review can change the document because the document is still analysis, not just a record.

The postmortems I come back to later are usually the ones that make the failure mechanism clear without preserving every piece of motion around it. They show where the system gave responders weak information, where a guardrail was missing, where the blast radius became larger than expected, and where recovery depended on something fragile.

That is what I want the document to leave behind. The timeline is useful, but the durable output is the changed system: the guardrail that now stops the rollout, the signal that catches the failure earlier, or the recovery step that no longer depends on somebody remembering it under pressure.

About the author

Lovell Felix

Infrastructure and reliability engineer working on Linux platforms, configuration delivery, and deployment safety at fleet scale.

@lovellfelix

More notes