Last month I finally sat down and looked at what our secrets management in production actually looked like. Not the architecture diagram. Not the policy doc that says the right things. The real thing — the 55 CI/CD pipeline variables that every build quietly depended on. Twelve of them were sitting in plaintext. Thirty were unused, leftovers from projects that shipped and got forgotten. And not a single one had a rotation policy. One database password hadn't been changed in fourteen months, and it was cheerfully printed into build logs that anyone with pipeline access could read.

Nobody knew. Nobody checked. The pipelines worked, so nobody asked questions. That is the quiet horror of security work — it stays invisible until it isn't, and the moment it stops being invisible is usually a meeting where you are explaining how a fourteen-month-old plaintext credential ended up somewhere it never should have been.

The audit nobody asked for

I wasn't assigned this. I just got tired of not knowing. So I pulled every variable out of every pipeline and read them one by one. Fifty-five entries. Half of them meaningless — pointing at resources that no longer existed. And buried in the noise, twelve real, live secrets in clear text: database passwords, an API key, storage credentials. Things that, if leaked, don't get you a bad afternoon. They get you a breach report.

Here is what makes it worse than it sounds: these were not test pipelines. They were production, running for months, moving real data. The credentials worked, so the system was "fine." That is the trap — a plaintext secret gives you zero symptoms until the exact moment it gives you all of them at once.

Why a manual audit is already a lie

The usual answer to this is a quarterly security review. Someone logs into the console, checks IAM by hand, clicks through security groups one at a time, and types it all into a spreadsheet. By the time they save that file, it is already fiction. A new resource got spun up during the review. Someone modified a storage account. The audit is a photograph of a past that no longer exists.

Manual security review doesn't scale, and worse, it lies to you with a straight face. It hands you a green checkmark for a system that changed while you were checking it. If your security posture lives in a spreadsheet, you are already behind — because attackers don't wait for your quarterly review.

I hold a CEH certification. Not for the LinkedIn badge — I got it because I wanted to think the way the people who exploit exactly this kind of laziness think. And the honest truth is that plaintext secrets in a build variable are not some exotic attack surface. They are the first thing anyone looks for.

What I actually did

The cleanup itself was not glamorous, and it took about a week:

  • Everything went into Azure Key Vault. No secret survives as a raw pipeline variable anymore.
  • Managed identity authentication. The pipeline proves who it is and pulls what it needs — no long-lived credential sitting in a settings field.
  • The 12 plaintext secrets were encrypted and moved behind RBAC. Access is a decision now, not a default.
  • Automatic rotation configured. The fourteen-month password problem cannot exist again, because nothing lives long enough to become that.
  • The 30 unused variables were deleted outright. Dead config is attack surface with no upside.

I made the pipeline the auditor

Cleaning up once solves the problem for exactly one day. The point is to make the bad state impossible to reintroduce. So I wired the checks into the pipeline itself, as hard gates — not warnings, actual blockers that stop the build:

  • Terraform compliance checks run before anything deploys, catching misconfigurations at the source instead of after they are live.
  • Secret scanning on every commit. A credential in a diff never reaches main.
  • AWS Security Hub centralizes findings across accounts, and custom Python scanners cover the compliance rules that don't fit off-the-shelf tooling.

Open security groups, overly permissive IAM, unencrypted storage, public buckets, missing logging — all caught automatically, in minutes, not once a year when an auditor visits.

The line I keep coming back to

A vulnerability without a remediation path is just anxiety. A vulnerability nobody is even looking for is a countdown.

Fifty-five variables. Fourteen months. Nobody looked — until I looked. That is the entire difference between a security incident and a Wednesday-afternoon cleanup. So let me hand you the same uncomfortable question I handed myself: when was the last time someone actually audited your pipeline variables?