It's 3 AM. The pager goes off. Production is down. And every instinct in your body is screaming the same thing: do something. Restart the service. Roll back the deploy. Change that config. Anything, now.
That instinct is wrong. In the incident response first 5 minutes, the fastest hands lose. I've watched a panicked engineer turn a 30-minute blip into an 8-hour outage more times than I can count — and early in my career, I was that engineer. What actually saves you at 3 AM isn't speed. It's a protocol you rehearsed while things were boring.
Here's the four-step framework I run on-call, every time, in order.
Step 1: STOP
Do nothing for 30 seconds. I mean it — hands off the keyboard.
Sounds insane when everything is on fire. But panic is a terrible engineer. It makes you restart the wrong service, and now two things are broken instead of one. Those 30 seconds are for the adrenaline to drain out of your fingers so your brain can come back online. The outage will still be there in half a minute. What you do next matters more than how fast you start.
Step 2: ASSESS
Figure out what is actually broken — not what it looks like.
500 errors are a symptom. "Database not responding" is closer. "Connection pool exhausted" is the cause. If you restart the app on a 500, you feel productive for four minutes and then it falls over again, because you treated the symptom. Trace the chain down to the thing that is actually failing before you touch anything. A symptom fixed is just an incident postponed.
Step 3: COMMUNICATE
This is the part nobody teaches you, and it's the one that builds or destroys trust.
Write one message. Not a paragraph, not a root-cause analysis — one line: "Investigating. Update in 10 minutes." That single sentence drops stakeholder anxiety by about 80% in my experience. Here's why it works: people can handle problems. What they cannot handle is silence while production burns. Silence makes them imagine the worst, then start pinging you individually — and now you're answering DMs instead of fixing the outage.
Bad news early beats good news late. Every single time.
Step 4: ACT
Now you act. And even here, discipline wins.
- One change at a time. Not three "just in case."
- Verify after each one. Did it help? Measure, don't hope.
- Didn't help? Roll it back immediately, before you try the next thing.
- Document every action as you go.
That last point is the sneaky one. The log of what you tried is your postmortem. Three days later, nobody remembers whether you bounced the pods before or after the config change. If you wrote it down live, you're not reconstructing the incident from memory — you already have the timeline.
Why communication beats speed
Here's the counterintuitive truth I keep coming back to: a well-communicated one-hour incident builds more trust than a silent twenty-minute fix that nobody understood.
The silent fix feels heroic. You went in, you crushed it, you came out. But your stakeholders lived through twenty minutes of not knowing whether the company was on fire. That fear doesn't evaporate because you fixed it fast. It compounds. The next time something breaks, they don't trust the system — or you.
The slow, narrated incident is the opposite. People saw you were on it. They got their 10-minute updates. They watched a professional work a problem. That's the reputation you actually want.
Rehearse it while it's boring
The engineers who stay calm at 3 AM all have one thing in common: they wrote their runbooks when nothing was wrong. Preparation is invisible right up until the moment it saves you.
If your postmortems keep ending with "restarted the service and it recovered," you don't have a process — you have luck, and luck runs out. Heroes burn out. Processes scale.
So practice this sober, so it works when you're panicked: STOP. ASSESS. COMMUNICATE. ACT. Save it somewhere you'll find it at 3 AM. Because you will need it at 3 AM.