Every prompt you send to ChatGPT, Claude, or Gemini runs on someone else's GPUs. Your code, your customer records, your company's secrets all travel through infrastructure you do not own and cannot audit. For most teams that is a fine trade. You call the API, pay per token, and move on. But if you work in healthcare, finance, or government, "fine" stops being good enough, and self-hosted AI on private data with a local LLM and your own GPUs becomes the only answer that survives a compliance review.
I want to be honest up front: most people reading this should just call an API. GPUs inside hyperscalers are expensive, the models improve every few weeks, and running inference yourself is genuinely more work. I'll say that again at the end, because it matters. But there is a real set of cases where self-hosting is not a hobby, it's a requirement.
Why self-host at all
Four reasons push a team off the managed API:
- Regulation. Data is legally not allowed to leave your environment. No exceptions, no "the vendor is SOC 2 certified" hand-waving.
- Control. Company policy forbids sending source code or production data to a third party.
- Cost at scale. Past a certain request volume, per-token pricing costs more than the hardware.
- Your own models. You've fine-tuned something on proprietary data. That model is intellectual property, and you are not shipping it to someone else's cluster.
There's a security angle people underrate too. Chinese open-weight models now account for roughly 30% of global AI usage. Security researchers found DeepSeek complied with 100% of malicious requests using common jailbreaking techniques. One hundred percent. Comparable US models fail somewhere between 5 and 12 percent of the same tests. Whatever you make of those numbers, self-hosting takes the data-exposure question off the table entirely. Your prompts stay on your servers. Nobody logs them, nobody trains on them, nobody leaks them.
The stack
The setup is boring in the best way, which is exactly what you want from infrastructure. A control plane cluster runs Crossplane with a set of custom API definitions I wrote. Crossplane is the declarative glue: instead of clicking through a cloud console, you describe the outcome you want as a Kubernetes resource and the platform reconciles reality to match it.
When someone applies a cluster resource, Crossplane does the whole dance:
- provisions a Kubernetes cluster with separate CPU and GPU node groups
- installs the Nvidia GPU operator to handle drivers and device management
- deploys vLLM for inference
- wires ingress, networking, and the supporting apps together
The person requesting it fills in a handful of fields: cloud provider, node size, a GPU toggle, and which apps to install. That is the entire surface they see. No Helm charts, no driver debugging, no GPU expertise required.
One resource, one endpoint
When a team needs a model, they apply a single custom resource. They specify the model, how many GPUs it gets, and the ingress host. That's it. vLLM exposes an OpenAI-compatible API out of the box, which is the quiet detail that makes the whole thing usable.
OpenAI-compatible means every tool already built for OpenAI just works. Point Cursor at your internal endpoint and get code completion on a model running in your own datacenter. Build agents with any SDK that speaks the OpenAI protocol. Swap the base URL, keep the code. Nobody has to learn a new client library, because there is no new client library.
Around it I built a 634-line universal CI/CD pipeline template that deploys everything from plain microservices to ML models through the same path. One pipeline, many workload types, no bespoke snowflake per team.
Your data, your GPUs, your rules
I deploy infrastructure under air raid sirens. Infrastructure control is not an abstract slogan to me, it's the thing that keeps systems running when everything around them is falling apart. That perspective is why the ownership question lands harder for me than the convenience one.
So here's the honest close, the same one I promised. If your workload is ordinary, call the API. It's cheaper, faster to start, and someone else carries the operational weight. But if regulations demand that data never leaves your environment, if policy forbids third-party inference, if your scale makes per-token pricing absurd, or if your fine-tuned models are the product, this is the setup. Local LLM, private data, your own GPUs.
If your AI prompts contain production data and run on someone else's GPUs, you don't have an AI strategy. You have a data leak you haven't noticed yet.
Where do your prompts actually run, and who else can read them?