Free template
Incident postmortem and root cause analysis template for software teams.
Fill in the incident, the key times, the timeline and a 5 whys chain. You get a blameless postmortem with time to detect, mitigate and resolve worked out, ready to paste into Confluence, Notion, Google Docs or a GitHub issue.
Filled with an example. Change anything, or start blank.
Postmortem: Checkout errors after a config deploy
Severity: SEV2 (Major) · Date: 2026-10-14 · Authors: Maya Chen, Tom Becker · Status: Draft
This is a blameless postmortem. It describes what the system and the process allowed to happen, not who made a mistake. Everyone involved acted on the best information they had at the time.
Summary
A config change set the payment provider timeout to 200 ms instead of 2000 ms. About a third of card payments timed out at checkout for 55 minutes until the change was rolled back.
Impact
About 38% of checkout attempts failed with a generic error between 09:00 and 09:55 UTC. Roughly 1,900 customers affected, about 420 orders lost or retried later. No data loss.
Services affected: checkout-api, payments-gateway
Key times
All times UTC.
| Time | ||
|---|---|---|
| Impact started | Oct 14, 09:00 | |
| Detected | Oct 14, 09:12 | Time to detect: 12 min |
| Mitigated | Oct 14, 09:55 | Time to mitigate: 43 min |
| Resolved | Oct 14, 15:30 | Time to resolve: 6 h 18 min |
Customer impact lasted 55 min.
Timeline
| Time | Event |
|---|---|
| Oct 14, 08:58 | Config change to payments timeout merged and deployed to all regions. |
| Oct 14, 09:12 | Checkout error-rate alert fires. Jordan (on-call) acknowledges. |
| Oct 14, 09:20 | Incident declared SEV2 in #incidents. Maya is incident commander. |
| Oct 14, 09:41 | Payment provider timeouts spotted in traces. Config change suspected. |
| Oct 14, 09:55 | Config rolled back. Error rate back to normal. |
| Oct 14, 15:30 | Config validation added and corrected value deployed. Incident resolved. |
Root cause analysis: 5 whys
Problem: Card payments timed out at checkout for 55 minutes.
- Why? The payment provider timeout was set to 200 ms instead of 2000 ms.
- Why? A config change was deployed with a value in the wrong unit.
- Why? The config file has no units and nothing checks the value's range.
- Why? Config changes skip the canary stage that code changes go through.
- Why? Config deploys were set up as a fast path years ago and never reviewed.
Root cause: Config deploys were set up as a fast path years ago and never reviewed.
Contributing factors
- Error alert only fired at a 25% error rate
- The runbook for checkout errors didn't mention config changes
- Two of four payments engineers were out that week
What went well
- Rollback took under 2 minutes once we knew the cause
- Status page updated within 15 minutes of detection
Where we got lucky
- It happened mid-morning on a weekday, not during a sale
Action items
| Priority | Action | Owner | Due | Ticket |
|---|---|---|---|---|
| P1 | Validate config values and units in CI | Tom | 2026-10-28 | |
| P1 | Route config changes through the canary stage | Maya | 2026-11-11 | |
| P2 | Lower the checkout error alert threshold to 5% | Jordan | 2026-10-21 | |
| P3 | Add config changes to the checkout runbook | Priya | 2026-10-31 |
Copy pastes as formatted text into Google Docs, Notion or Confluence. Markdown is for GitHub, Jira or Slack.
Paste it into Confluence, Notion, Google Docs, a GitHub issue or your incident tool. Nothing you type leaves your browser.
Incident severity levels, SEV1 to SEV4.
Agree the levels before you need them, so nobody argues about severity at 3 am. Adjust the examples to your product, and keep the number of levels small.
| Level | Definition | Examples | Response | Updates | Postmortem |
|---|---|---|---|---|---|
| SEV1Critical | The product is down or a core flow is broken for most customers, data is lost or exposed, or there is a security breach. | Checkout fails for everyone; login is down; customer data leaked; production database lost writes. | Page on-call at once, any hour. Incident commander named in minutes. Pull in whoever is needed. | Status page and stakeholders every 30 minutes. | Required, within 5 working days. |
| SEV2Major | A core feature is broken or badly degraded for many customers, or one large customer is down. A workaround may exist. | Payments slow for 20% of users; mobile app crashes on launch for one OS version; webhooks delayed by an hour. | Page on-call at once, any hour. Incident commander named. | Status page and stakeholders every hour. | Required, within 5 working days. |
| SEV3Minor | A non-core feature is broken or some customers see a degraded experience. A workaround exists. | CSV export fails; search results stale for an hour; one region has higher latency. | On-call handles it in working hours. No page at night. | In the incident channel when status changes. | Optional. Write a short one if the team can learn from it. |
| SEV4Low | A cosmetic issue or an internal problem with no customer impact yet. | Typo in an error message; an internal dashboard is down; an alert is noisy. | A ticket in the backlog. | None needed. | Not needed. Track the fix as a normal ticket. |
When in doubt, pick the higher level. It's easy to downgrade an incident and awkward to explain why you didn't page anyone.
Blameless postmortems, and why they find more.
Ask how, not who
“How did a bad config reach production?” finds the missing check. “Who deployed it?” finds a person, who will be more careful for a month, and the gap stays.
Assume good intent
Everyone acted on what they knew at the time. If the decision looks wrong now, ask what made it look right then: the dashboard, the runbook, the time pressure.
People are not root causes
If an answer is “someone made a mistake”, keep going: why was the mistake possible, and why wasn't it caught? Fix the system so the next person can't make it.
Name actions, not people
Write “the deploy ran without a canary”, not “Tom skipped the canary”. People can be named in the timeline for what they did, never as the cause.
Share it widely
A postmortem nobody reads teaches one team. Post it where other teams will see it, and run a short read-out for SEV1s.
Reward the reporter
Thank whoever raised the alarm or owned up to the change. If speaking up gets punished, the next incident is found by customers.
Root cause analysis with the 5 whys.
Write the problem as a fact with a number in it: “Card payments timed out at checkout for 55 minutes.” Then ask why, and why again. Each answer should be something you could check in the logs, the code or the config. In the example above the chain runs from a wrong timeout value, to a value in the wrong unit, to a config file with no units and no range check, to config changes skipping the canary. The last two are what you fix.
Stop when you reach something the team can change in the next few sprints. Going further usually ends at “we had too much to do”, which is true and not actionable. Go back when an answer names a person, and ask why the system let it happen.
Five whys finds one chain, and most incidents have several. Put the others under contributing factors: the alert threshold that was too high, the runbook that didn't mention config, the two engineers who were out that week. For a problem with many causes, run a fishbone exercise in the meeting instead.
How to run the postmortem meeting.
Write the draft first, then meet. The meeting is for checking the story and agreeing the actions, not for writing the document together.
- BeforeThe incident commander or on-call writes the draft within two days: timeline from the incident channel, alerts and deploy log, impact numbers, a first 5 whys. Share it a day before the meeting.
- 5 minSet the frame. Read the blameless line out loud. Say what the meeting is for: agree what happened and what we'll change.
- 15 minWalk the timeline. People who were there correct and fill gaps. Look for where time was lost: before the alert, before the right person joined, before the cause was found.
- 15 minRoot cause and contributing factors. Check the 5 whys chain. Add factors people raise. Ask what went well and where you got lucky.
- 15 minAction items. For each: does it prevent a repeat, detect it sooner, or mitigate it faster? Keep three to five, each with one owner and a due date.
- 5 minClose. Agree who publishes the final version and where. Book a 15 minute check on the actions in four weeks.
Invite the people who responded, the owner of the affected service and someone from support if customers noticed. Keep it under an hour. Managers who weren't involved can read the document.
Making action items actually happen.
Fewer, smaller actions
Three done beats ten in the backlog. Split “improve monitoring” into “alert on checkout error rate above 5%”.
One owner, a real date
A team is not an owner. A due date of “Q1” is not a date. If nobody will own it, it's not an action, it's a wish.
Tickets, not a doc
Put each action in the tracker with a postmortem label and link back. Bring P1s into the next sprint planning.
Review them in public
Check open postmortem actions in a weekly ops review or at the start of each retro. Overdue P1s go to the engineering manager.
Close the loop
When the last action is done, update the postmortem and say so in the channel. It shows the process works.
Drop them on purpose
If an action no longer makes sense, close it with a reason. Silent rot teaches people that postmortems don't matter.
Incidents over the holidays, when on-call is out.
Many postmortems have the same contributing factor: the person who knew the system was away, or the on-call shift fell on someone's vacation. On this board, Tom's on-call week lands on his trip home for Christmas, inside a holiday code freeze. Swap shifts before the holidays, not during the incident.
- On-call shifts next to time off
- A shift that clashes with a vacation is flagged
- The holiday freeze on the same board
- Drag a shift to another person
Before a holiday period, check that every shift has a primary and a backup who are actually around, that the runbooks for your SEV1 scenarios are current, and that someone besides one person can roll back each critical service. The on-call schedule generator builds a rotation with the holidays in each shift, and the bus factor calculator shows which systems only one person can handle.
Questions
What should an incident report include?
A title and severity, a short summary, the customer impact, the key times (when impact started, was detected, mitigated and resolved), a timeline of what happened, the root cause and contributing factors, what went well, and action items with owners and due dates.
What is a blameless postmortem?
A postmortem that looks at how the system and the process let an incident happen, not at who made a mistake. It assumes people acted reasonably with what they knew at the time, so they can describe what really happened without fear, and the fixes go into tools and process.
What are incident severity levels?
A shared scale for how bad an incident is, usually SEV1 (critical: product down or data at risk) to SEV4 (low: cosmetic or internal). The level sets who gets paged, how often stakeholders hear from you, and whether a postmortem is required.
How do you use the 5 whys for root cause analysis?
Write the problem as a fact, ask why it happened, then ask why that answer happened, about five times. Stop when you reach something the team can change. If an answer is a person's mistake, ask why the system allowed it. Most incidents have more than one chain, so list contributing factors too.
What is the difference between an incident report and a postmortem?
An incident report records what happened: impact, times and timeline, often written during or right after the incident. A postmortem adds the analysis: root cause, contributing factors and action items. Many teams use one document for both, which is what this template does.
When should you write a postmortem?
For every SEV1 and SEV2, within about five working days while memories are fresh. For SEV3, write a short one when there is something to learn. Also write one for near misses, like a bad deploy caught before customers noticed.