When you set out to migrate VMs to GKE Kubernetes, the hard part is almost never the Kubernetes. The hard part is the sentence you hear on day one: "It works. Don't touch it."
That was the philosophy on this team. Multiple production services, each running on its own virtual machine. Every one a snowflake — hand-configured, patched live, impossible to reproduce. If it ran, you left it alone. So I asked one question in a standup: what happens when one of these VMs dies on a holiday and nobody remembers how it was set up? Silence. That silence was the whole project.
"It works" is not a resilience strategy
Let me describe what "it works" actually looked like. Deploying meant SSH into the box, pull the latest code, restart the process, cross your fingers. A full deployment cycle took hours. There was no horizontal scaling — just over-provisioned VMs burning money around the clock so they could survive a traffic spike that might come twice a year. And when a machine died, we rebuilt it from scratch, from someone's half-remembered notes about how it was configured months ago.
None of that shows up on an uptime dashboard. That's the trap. Snowflake VMs are single points of failure hiding behind good statistics, and nobody wants to hear that — especially not from the new person. The real resistance was never technical. It was about change, and about admitting the current setup was fragile.
Step one was not Kubernetes. It was containers.
Here's the mistake people make: they treat "migrate to GKE" as the first step. It isn't. If you can't reproduce a service on your laptop, a cluster won't save you — it'll just spread the chaos across more nodes.
So step one was containerization, and it had nothing to do with Kubernetes yet:
- Multi-stage Dockerfiles so the build environment never leaks into the runtime image.
- Minimal base images — smaller attack surface, faster pulls, fewer surprises.
- Every image versioned and stored in Google Artifact Registry, so any deploy is traceable to an exact artifact.
- Environment parity across dev, staging and prod. That was the entire point — a container is a contract that says this application runs the same everywhere.
The stateful services fought us hardest, as they always do. Anything that assumed a permanent local disk and a machine that never moved had to be rethought. That work was slow and unglamorous, and it was also where most of the real value lived.
GKE architecture: not deploy-and-hope
Only once the foundation was solid did GKE come in. And a real cluster looks nothing like the tutorial one:
- Node pools segmented by workload type — compute-heavy services separated from I/O-bound ones, so they stop competing for the same resources.
- Horizontal pod autoscaling on real application metrics, not just CPU. CPU is a lazy proxy; queue depth and request latency are what actually tell you to scale.
- Network policies for service-to-service isolation, so a compromised pod can't talk to everything.
- Sensible resource requests and limits, so one noisy service can't starve the rest.
What actually changed
- Deployment time went from hours to minutes.
- Infrastructure cost got right-sized by autoscaling instead of paying for peak load 24/7.
- Zero snowflake servers. Every service is now reproducible from a Dockerfile and a Helm chart.
- When a node dies, Kubernetes reschedules the pods. The 3 AM panic calls and the rebuild-from-memory rituals are gone.
Kubernetes was never the goal
This is the part I keep coming back to. Kubernetes is not cool technology you adopt to sound modern. Containers aren't cool either. A container is a contract — this app will behave the same on my laptop, in staging, and in production. GKE is just the thing that keeps that contract honest at scale and reschedules the work when hardware fails. The goal was never Kubernetes. The goal was running software reliably, without heroics.
The best part? The team that fought the migration hardest now runs the kubectl trainings for new hires. The people most attached to "don't touch it" became the loudest advocates once they felt what self-healing infrastructure does to their on-call weeks. I couldn't make that up if I tried.
What was the migration your team resisted most — and did they come around?