Ad · 728×90 desktop · 320×50 mobile

What happens after a P1 incident (and what should)

Published 2026-04-24 · SRE & Observability

An incident ended. The site's back up. Now what?

What usually happens

  • 24 hours: relief
  • 48 hours: a "postmortem" meeting that's mostly recap
  • 1 week: action items documented
  • 1 month: 40% of action items done, 60% forgotten
  • 6 months: same class of incident happens again

What should happen

Hour 0: stabilize

Fix the immediate issue. Mitigate customer impact. Communicate.

Hour 12: timeline

Before details fade, write the timeline of events. All times UTC. Specific actions taken.

Day 2-3: five whys (or equivalent)

Why did this happen? Why did THAT happen? Keep going. Stop when you reach a systemic cause, not an individual action.

Day 3-5: draft postmortem

Include: impact, timeline, root cause, what went well, what didn't, action items with owners and dates.

Day 7: review meeting

Reviewed by: on-call engineer, service owner, engineering lead, someone from a different team (fresh perspective). Facilitator runs the meeting.

Day 14: published

Publish broadly. If customer-facing incident, often publish a customer-facing version too.

Day 30: action item status

Review action items. Completed? In progress? Dropped (and why)?

Quarter review

Look at last 90 days of incidents. Patterns? Action item completion rate?

The hard rules

Blameless

The humans made reasonable decisions given what they knew. The system failed them, not vice versa.

Action items MUST be completed

If action items drop, owners pick new ones. The item stays open until resolved.

Root cause is systemic, not individual

"Engineer forgot to run migration" is not a root cause. "Deployment process doesn't enforce migration ordering" is.

Tools

Ad · 300×250 responsive

Joined by engineers at enterprises running the systems you build. No spam. No tracking in the emails.