What happens after a P1 incident (and what should)
An incident ended. The site's back up. Now what?
What usually happens
- 24 hours: relief
- 48 hours: a "postmortem" meeting that's mostly recap
- 1 week: action items documented
- 1 month: 40% of action items done, 60% forgotten
- 6 months: same class of incident happens again
What should happen
Hour 0: stabilize
Fix the immediate issue. Mitigate customer impact. Communicate.
Hour 12: timeline
Before details fade, write the timeline of events. All times UTC. Specific actions taken.
Day 2-3: five whys (or equivalent)
Why did this happen? Why did THAT happen? Keep going. Stop when you reach a systemic cause, not an individual action.
Day 3-5: draft postmortem
Include: impact, timeline, root cause, what went well, what didn't, action items with owners and dates.
Day 7: review meeting
Reviewed by: on-call engineer, service owner, engineering lead, someone from a different team (fresh perspective). Facilitator runs the meeting.
Day 14: published
Publish broadly. If customer-facing incident, often publish a customer-facing version too.
Day 30: action item status
Review action items. Completed? In progress? Dropped (and why)?
Quarter review
Look at last 90 days of incidents. Patterns? Action item completion rate?
The hard rules
Blameless
The humans made reasonable decisions given what they knew. The system failed them, not vice versa.
Action items MUST be completed
If action items drop, owners pick new ones. The item stays open until resolved.
Root cause is systemic, not individual
"Engineer forgot to run migration" is not a root cause. "Deployment process doesn't enforce migration ordering" is.