We have built AI products on a single container service, on a full Kubernetes platform, and on most things in between. The technology rarely decided the outcome. The match between the platform and the team that had to run it did. This post is about the mismatch that cost a client six months, the questions that avoid it, and the shape we now recommend for most AI products.
The platform that was too big
A SaaS client with three engineers had adopted Kubernetes because the AI feature they planned "would need to scale." A year later they had a cluster, a service mesh, a custom deployment pipeline, and an AI feature that had not shipped, because every week contained a platform problem. Upgrades, certificate rotation, a networking issue that took a fortnight, and a cloud bill for capacity that sat idle waiting for the scale that had not arrived.
We migrated the workloads to managed container services and a managed queue, kept the same containers, and removed the cluster. The AI feature shipped the following quarter. Nothing about the product needed the platform; the team's time had been the constraint all along.
Infrastructure should be sized to the team that operates it, not to the traffic in the pitch deck.
The questions that decide it
How many services, and how different are they? A handful of web services and workers run well on managed container platforms. Dozens of services with complex networking and custom scheduling are where an orchestrator starts to earn its keep.
What needs a GPU, and when? Model inference through a provider API needs no GPUs at all. Self-hosted models need them, and the question becomes whether the load is steady enough to justify reserved capacity or bursty enough to prefer serverless GPU offerings. Training jobs are batch: they want a job runner, not a cluster kept warm.
Who is on call? If the answer is the same three people who build the product, every operational surface you add is product time you remove.
What does the security team need? Network isolation, secrets management, audit logs and image scanning are available on managed services. They are also available on Kubernetes, with more assembly. The requirement is the same; the effort is not.
The shape that usually fits
For most AI products at mid-size scale, the shape looks like this. Web and API services on a managed container platform with autoscaling. Background work, including document ingestion and model calls that can be asynchronous, on a managed queue with worker containers. Inference through a provider API, or a managed model endpoint when data must stay inside the boundary. Batch jobs, including retraining and evaluation runs, on a managed job runner. A managed database and object storage. All of it defined in infrastructure-as-code, deployed by a pipeline that runs tests and the evaluation gate before anything reaches production.
This runs comfortably with a small team, scales further than most products will need, and leaves a clear migration path to an orchestrator if and when the service count and traffic justify it.
Infrastructure as code is not optional
Whatever the size, the environment is defined in code and applied by a pipeline. This is what makes environments reproducible, changes reviewable, and security policy enforceable: network rules, identity and access, encryption settings and logging all live in the same repository as the application, reviewed in the same pull requests. A platform that was clicked together in a console cannot be audited, cannot be rebuilt, and cannot be trusted by a security team, however small it is.
Security specifics for AI workloads
AI workloads add a few requirements to the usual list. Secrets for model providers need rotation and scoping per environment. Egress to providers should be explicit and logged, so that data leaving the boundary is a known, monitored path. If models are self-hosted, model artefacts are treated as sensitive assets with controlled storage. Inference endpoints get authentication, rate limits and cost budgets like any API. And the pipeline that promotes a model is a deployment pipeline, subject to the same review and approval as application code.
Cost, which follows from the shape
Right-sized platforms are cheaper mostly because they run less idle capacity and consume less engineering time. Managed services cost more per unit than raw compute and far less per unit of team attention. For GPU workloads, the difference between reserved capacity kept warm and on-demand capacity for actual load is usually the largest single line on the bill, and it is decided by the shape, not by negotiation.
Purposefully sized
Choose the platform the team can run well today, define it in code, and grow it when the product proves it needs more. Kubernetes is a fine answer to a real question. It is a poor answer to "we might need to scale," which is a question every product asks and most never need to answer.
- Count services, GPU needs, and the people on call.
- Use managed services for web, workers, queues, jobs, databases and storage.
- Call models through a provider or a managed endpoint before self-hosting.
- Define everything in infrastructure as code with a reviewed pipeline.
- Treat provider egress, secrets and model artefacts as security assets.
- Grow into orchestration when the service count demands it, not before.