"You put GPT-4 WHERE?"

That was my manager's face when I proposed putting Azure OpenAI in a production pipeline. Not a sandbox. Not a proof of concept that lives in a Jupyter notebook and dies on Monday. Production. Enterprise e-commerce. Real customers spending real money, on real infrastructure, at real scale.

Everyone is rushing to bolt AI onto something right now. Very few are thinking about how to ship it safely. That gap is the whole story.

What the AI actually does

Once it was live, GPT-4 earned its keep in four places:

  • Intelligent code review — it catches the patterns humans miss at 2 AM, on the fifth review of the night when your eyes have glazed over.
  • Incident classification — it reads the alert and routes it to the right team in seconds instead of a human triaging by hand.
  • Natural language infrastructure queries — "show me all pods with memory above 80%" instead of memorizing kubectl flags.
  • Deployment risk scoring — it flags the changes that historically caused trouble before they go out.

None of that replaced a single engineer. The junior who used to burn 30 minutes reconstructing a kubectl command now asks in plain English and gets an answer in seconds. The on-call engineer who used to hand-classify every page now has a co-pilot that routes about 80% of incidents automatically. Zero headcount cut. Faster response times. Fewer human mistakes. That's the AI story nobody writes, because "we quietly made the team better" is a terrible headline.

The pipeline is the product

Here's the part the demos skip: the model was the easy part. The pipeline around it was the actual engineering. Four things had to be true before anything reached a customer.

  • Versioned prompt templates. A prompt change is a deployment change. You do not push a new prompt to production just because it looked good in the chat window. You version it, review it, and test it like code.
  • An API gateway in front of the model, with rate limiting and authentication, so one runaway loop can't melt your budget or your quota.
  • Integration tests that validated answer quality, not just a 200 OK. An HTTP status tells you the model replied. It tells you nothing about whether the reply was right.
  • Canary releases with automatic rollback. If the AI starts giving wrong answers to paying customers, you revert in seconds, not hours.

On top of that, KQL dashboards tracked token usage, response latency, error rates, and cost per request in real time. That observability layer took our AI deployment time down by about 60% — from manual, nerve-wracking releases to something automated, observable, and repeatable.

The hard lessons

Production teaches you things no prompt-engineering blog will:

  • Prompt changes are deployment changes. Version them. Test them. Review them like code. Skip this and you'll ship a "small wording tweak" that quietly breaks 10% of your responses.
  • Cost can explode overnight. Set hard token limits on day one, not after the first surprise invoice. AI spend doesn't fail loudly — it just grows.
  • Latency budgets matter more here. GPT-4 is not fast. Plan for it, set timeouts, and keep a fallback for when the model is slow or down.
  • Rollback is not optional. When wrong answers cost real money, seconds matter.
  • Human-in-the-loop is not a nice-to-have. It is the architecture. The one time the AI confidently recommended deleting a production database — full confidence, zero hesitation — a human caught it. That's not a funny anecdote. That's the design working exactly as intended.

The uncomfortable ratio

If I had to compress it: production AI is about 20% prompt engineering and 80% error handling, fallbacks, monitoring, and knowing when NOT to call the model at all. The interesting, tweet-worthy work is the small slice. The boring DevOps work is everything else — and it's what keeps you out of the incident channel.

AI without DevOps is a science experiment. AI with DevOps is a product. Build the pipeline first — the model really is the easy part.

So here's the question I keep coming back to: how do you handle the moment when the AI hands you a confident answer and your gut says something's off? Because that moment, not the model, is where production AI actually lives.