Projects

Production Is Down. 3 AM Alert. Your First 5 Minutes Decide Everything.

3 AM. Alert. Production is down. Most engineers at this moment make a mistake that turns a 1-hour incident into an 8-hour one. Step-by-step guide for the first 5 minutes: STOP (don't touch anything for 30 seconds), ASSESS (symptoms vs root cause), COMMUNICATE (one message reduces anxiety by 80%), ACT (one change at a time, verify each). Save this video. You'll need it at 3 AM. The instinct to start changing things immediately is the single most dangerous move when production is on fire. Panic makes you restart the wrong service and now two things are broken. The four-step framework here is what I actually use on-call: stop for 30 seconds to let the adrenaline pass, assess what's actually broken - 500 errors are a symptom, connection pool exhausted is the cause, treat the cause not the symptom, then communicate. One message in the channel: "Investigating. Update in 10 minutes." That single line reduces stakeholder anxiety by 80%. People can handle problems. They cannot handle silence while production burns. Only then do you act. One change. Verify. Rollback if it didn't help. Document every action so you know what you tried. A well-communicated one-hour incident builds more…