3 AM. CEO on Slack. Production Down. What I Did in the Next Hour Saved My Job.
3:14 AM. PagerDuty screaming. Production completely down. Database connection pool exhausted. API returning 500s. Payments failing. Customers tweeting. I check Slack. CEO is typing. Heart rate: 140. This is the story of the worst on-call incident of my career - and how the 5-minute protocol I used turned an 8-hour disaster into a 47-minute resolution. And a promotion. The scene: database connection pool exhausted, API cascading 500s, payments failing, CEO in Slack - "What's happening? Major client just texted me." Every instinct said start changing things. Restart services. Scale up. Anything to make it stop. That's the mistake. I've watched engineers turn a 1-hour outage into an 8-hour catastrophe by doing exactly that - no plan, no documentation, no communication. The 5-step protocol I actually used: Step 1 - STOP. Don't touch anything for 30 seconds. Let the adrenaline pass its first peak. Step 2 - ASSESS. 500s are a symptom. Connection pool exhausted is closer to the cause. Step 3 - COMMUNICATE. One message in Slack: "I'm investigating. Update in 10 minutes." That single message reduced CEO anxiety by 80%. Step 4 - DOCUMENT. Every command logged. Step 5 - ACT. One change…