Every dashboard was green. "All systems operational." And the whole time, our monitoring was quietly lying to me.

The support queue told the real story. Customers reporting errors. Pages crawling. Transactions timing out. None of it showed up in our metrics, because our green dashboards were hiding real failures behind a wall of numbers that looked reassuring and meant nothing. That gap — between what we measured and what users actually felt — is why I eventually tore the whole thing down and rebuilt monitoring from scratch.

We measured what was easy, not what mattered

It wasn't sabotage. It was the path of least resistance. We measured what was easy to measure: CPU, memory, disk space. Infrastructure metrics. The stuff every default Prometheus exporter hands you for free.

What we never measured was the thing that actually mattered: transaction success rates, end-to-end latency, error rates broken down by customer segment. The user's experience. So the servers could hum along at 30% CPU while a checkout flow silently threw 500s, and every panel on the wall stayed a calm, confident green.

Prometheus and Grafana are the classic example of this. Everyone installs them. Almost nobody sets them up right. The mess I keep finding is always the same:

  • Prometheus scraping everything and alerting on nothing useful.
  • Grafana dashboards built during setup week and never opened again.
  • No retention policy, so the disk fills up and monitoring dies silently — the monitor needs a monitor.
  • A single Prometheus instance that goes down with the very application it's supposed to watch.

47 dashboards, 200 alerts, zero trust

By the numbers, our setup looked serious. 47 dashboards. Over 200 alerts. It looked like a team that cared.

Nobody looked at 44 of those dashboards. Nobody responded to the alerts, because there were too many of them and each carried too little context. That's alert fatigue: when everything pages you, nothing does. Green stopped meaning "everything is fine." It meant "I hope everything is fine."

The most unsettling discovery came when I actually checked what was reporting. Large parts of the fleet weren't sending data at all. They were simply silent, and silence rendered as green. A dashboard that shows green while most of your fleet is invisible is worse than no dashboard, because at least with nothing you know you're blind.

Tearing it down

So I rebuilt it, and I started from philosophy, not tooling.

  • High-availability Prometheus with federation. Not one hopeful instance, but replicated collection with clear ownership per service team.
  • Grafana dashboards provisioned as code. No more hand-crafted masterpieces that vanish the moment someone clicks delete.
  • User-centric metrics first, infrastructure second.

The dashboard count went from 47 to 5. Each of the five tells a story instead of dumping graphs. If a dashboard didn't answer a question someone actually asks during an incident, it didn't survive.

Alerting that respects the human on call

Alerts went from 200-plus to 12. Every one survives on a single rule: if nobody acts on it, it gets deleted. I split them into tiers with honest definitions:

  • Critical: someone is losing money right now.
  • Warning: something will break by morning if we ignore it.
  • Info: goes to a dashboard and never pages a human.

And every single alert links to a runbook. Waking someone at 3am without telling them what to do next isn't monitoring, it's cruelty. Alert on the symptoms a user feels, not on causes that may or may not matter.

The hardest conversation

The hardest part wasn't technical. It was walking into leadership and saying: "Our 99.9% uptime metric is technically accurate and completely misleading."

The server was up. The application was broken. Those are two very different things, and our headline metric couldn't tell them apart. Nobody enjoys hearing that the number they've been quoting to customers is a comfortable fiction. But a metric that makes everyone feel good while the product falls apart isn't a metric worth keeping.

Monitoring is a product

The mindset shift that made all of it stick: monitoring is a product, and it has users. Its users are the on-call engineers at 3am. They deserve the same care you'd give a paying customer — clear stories, low noise, an obvious next step.

If you measure everything, you measure nothing. Measure what matters, and show the real picture even when the real picture makes people uncomfortable.

So here's the uncomfortable question. If your dashboard is green right now and your users are complaining, it isn't reassuring you. It's lying to you.