Every AI product we have shipped has changed model provider at least once. Sometimes for price, sometimes because a better model appeared, once because a provider's terms changed in a way the client's legal team would not accept. The teams that handled it in a week had designed for it. The teams that took a quarter had a provider's name written into every corner of the system. This post is about what "designed for it" means in practice.
The week and the quarter
The quarter-long switch was a system in which prompts had been tuned, by hand, to one model's quirks; provider-specific features were used directly in application code; there was no evaluation set, so nobody could say whether the new model was as good; and the cost model assumed one pricing structure. Switching meant rewriting prompts blind, refactoring every call site, and discovering quality problems from user complaints.
The week-long switch was a system with a model gateway, a versioned prompt library, an evaluation set and a routing layer. The new provider was added to the gateway, the evaluation set ran against it, three prompts were adjusted where the results differed, the routing was flipped for a small share of traffic, then all of it. The application code did not change.
Provider choice should be a configuration decision backed by an evaluation run, not a rewrite.
The four pieces
A model gateway. One internal interface for "send this prompt, with these tools, get a response," with provider adapters behind it. Application code never names a provider. The gateway also centralises logging, cost tracking, rate limiting, retries and the tenant-aware routing that multi-tenant products need.
Versioned prompts, separate from code. Prompts live in a library with versions, and each version records which models it was evaluated against. A prompt that works on two providers is preferred over one that is tuned to a single model's habits, even if the tuned one scores slightly higher, because the difference is the cost of the next switch.
An evaluation set that runs per provider. The same set described in our post on evaluation, run against each candidate model, produces a like-for-like comparison of quality, refusal behaviour, cost and latency. Without it, a switch is a leap.
A routing layer. Requests are routed by task type, tenant requirements, cost budget and health. This is what lets a switch happen gradually, what sends small tasks to small models, and what fails over when a provider has an outage.
Where lock-in hides
The obvious lock-in is the API. The less obvious kinds cost more. Provider-specific features, such as a particular structured-output mode or a proprietary tool-calling format, that the application depends on directly. Embeddings: an index built with one provider's embedding model cannot be queried with another's, so a switch means re-embedding everything, which must be planned and budgeted. Fine-tuned models, which exist only on the provider that trained them. Prompt caches and provider-side storage of files and conversations, which have to be recreated. And the quiet one: evaluation results that were only ever produced on one model, so nobody knows what "as good" means.
The rollout that makes it safe
A provider switch is a deployment and it gets a deployment's discipline. The evaluation set passes first. Then a small share of traffic routes to the new model, with the traces compared against the old model's on the same requests. Then a larger share. At every step the routing can be flipped back in seconds. Users are not told, because if the switch is done well there is nothing to tell them.
Security and contractual portability
Provider changes are often driven by terms, so the terms are part of the design. Data-handling requirements, retention, region and training-use exclusions are written down once, as requirements, and each provider is assessed against them before it is added to the gateway. When a provider's terms change, the assessment is rerun and the routing decision follows. Keeping this as a checklist rather than a memory is what let one client move quickly when they needed to.
Purposeful independence
None of this is an argument against using a provider's best features. It is an argument for using them through a layer that knows they are provider features, with an evaluation set that can say what you would lose by moving. The providers will keep changing. The systems that keep working are the ones that treated that as a certainty rather than a surprise.
- Put a gateway between the application and every provider.
- Version prompts and record which models they were evaluated on.
- Run the evaluation set per provider; compare quality, refusals, cost, latency.
- Route by task, tenant, budget and health; switch gradually.
- Plan for the hidden lock-ins: embeddings, fine-tunes, provider-side state.
- Write the data-handling requirements once and assess every provider against them.