A model is not finished when it ships. It is finished when it is retired. Everything between those two points is MLOps, and it is where most of the value and most of the risk lives. This post is for teams launching their first model or their first AI feature, written around the failure we see most: a system that worked at launch and slowly stopped, with nobody noticing.
The quiet failure
A retail client launched a demand forecast we had built together. It was accurate for the first two months. Then the business changed its promotion calendar, the input distribution shifted, and the forecast drifted. Nobody was alerted, because the pipeline ran green every day: it produced a number, on time, with no errors. The number was simply wrong. The planners noticed eventually, stopped trusting it, and went back to their spreadsheets.
The pipeline had monitoring. The model did not. Those are different things.
Infrastructure monitoring tells you the system ran. Model monitoring tells you whether it was right.
The four things to monitor
Inputs. Are the features the model receives still distributed the way they were at training time? A sudden change in a field's range or missing rate is the earliest warning.
Outputs. Has the distribution of predictions changed? A classifier that suddenly says "yes" twice as often has either found something or broken.
Outcomes. When the truth becomes known, how did the prediction compare? This is the only real accuracy measure, and it arrives late, so the first two exist to buy time.
Operations. Latency, cost per prediction, error rates, and for language models, token usage and provider health.
Retraining is a decision, not a schedule
Teams often ask how often to retrain. The honest answer is: when the monitoring says the model has degraded and there is enough new labelled data to fix it. A fixed schedule retrains models that are fine and misses the ones that broke on Tuesday.
What we set up instead is a retraining pipeline that can be triggered by a person or by a drift alert, that trains on a defined window of data, evaluates against a held-out set and the previous model, and only promotes the new version if it is better on the metrics that matter. Every version is stored with its training data reference, its evaluation results and who approved it.
Quality checks that catch what monitoring cannot
Some failures are not drift. A feature pipeline starts producing nulls after an upstream schema change. A label source begins recording a different definition. A language model provider updates their model and the outputs change tone. For these we run checks at each stage: data validation before training, evaluation gates before promotion, and for LLM features, a fixed evaluation set that runs against every prompt or provider change.
The operating model, which is the actual hard part
Tools are the easy part. The hard part is the roster: who looks at the drift dashboard, who decides to retrain, who approves a promotion, who answers when the business says the numbers look wrong. On the retail project the fix was not a better pipeline. It was a named owner on the client side, a weekly fifteen-minute review of the monitoring, and an agreed threshold at which we would retrain.
We now refuse to launch a model without that roster written down. It is the difference between a system and an experiment.
Security in the model lifecycle
Model systems carry their own security questions. Training data is often more sensitive than the model, so it needs the same access control and retention rules as the source systems. Model artefacts and prompts are intellectual property and should be versioned in a controlled store, not a shared drive. Prediction endpoints need authentication, rate limits and logging like any API. And for language models, every input from an external source has to be treated as untrusted content; the monitoring should include attempts to make the model do something it should not.
Improvement, purposefully
The point of monitoring is not to admire dashboards. It is to create a loop: outcomes reveal errors, errors become labelled examples, examples improve the next version, and the business sees the metric move. A model that has been in production for a year with that loop running is worth far more than the one that launched. One without the loop is worth less every month.
- Monitor inputs, outputs, outcomes and operations, separately.
- Retrain on evidence, not on a calendar.
- Gate every promotion with validation, evaluation and an approval.
- Version models, data references and prompts together.
- Write the roster: who watches, who decides, who answers.