The first month of an AI feature in production often produces two surprises. The first is that people use it more than expected. The second is the invoice. Language-model costs scale with usage and with the length of what is sent and received, and a design that was fine for a pilot of twenty users can become uncomfortable at two thousand.

This post is about the client whose bill tripled in a month, the four levers we used to bring it back down, and the way we now design cost in from the start.

What was actually expensive

The feature was a document assistant for an internal team. When we looked at the usage logs, the cost was not where anyone expected. Most requests were cheap. A small share were enormous: users pasting entire documents into the chat, follow-up questions that re-sent the whole conversation history each turn, and a "summarise everything" button that pulled forty documents into a single prompt. On top of that, every request used the largest model available, including requests to reformat a date.

Cost is a distribution, not an average. Find the tail before you optimise the middle.

Where the spend actually went: a small share of requests carried most of the cost.
Where the spend actually went: a small share of requests carried most of the cost.

Lever one: send less

The largest saving came from sending less text. Conversation history was trimmed to the turns that mattered, with older context summarised. Retrieval returned the passages that answered the question rather than the whole document. The summarise-everything button was replaced with a summary that was generated once when documents were ingested and stored, so the expensive work happened one time rather than on every click.

Lever two: route to the right model

Not every request needs the most capable model. We classified requests by difficulty: formatting, extraction and classification went to a small fast model; open-ended reasoning and drafting went to the large one. The classifier itself was a tiny model or, for many cases, a rule. Quality on the evaluation set did not move. Cost on the routed requests fell by most of the difference between the two models' prices.

Lever three: cache what repeats

The same question is asked more often than intuition suggests, and the same documents are retrieved for many different questions. We cache at three levels: exact repeated requests, the retrieval results for a query, and where the provider supports it, the shared prefix of a prompt so that the fixed instructions and the document context are not re-billed on every turn. Cache hits also return faster, which users noticed before finance did.

Lever four: budgets that stop runaway use

Every user, session and feature now has a budget: a cap on tokens per request, steps per agent session, and spend per day, with a clear message when it is reached. This is a security control as much as a cost control. An injection that loops the model, or a bug that retries forever, is bounded before it becomes an invoice.

The four levers in the order we apply them, and the design questions each one raises.
The four levers in the order we apply them, and the design questions each one raises.

Cost as a design property

After that project we changed how we plan. Cost per request is now estimated during design for the expected and worst-case inputs. The evaluation set carries a cost and latency budget alongside quality thresholds, so a prompt change that improves answers but doubles tokens is visible before it ships. Dashboards show cost by feature, by user group and by request tail, not just the monthly total.

The important shift is who owns it. Cost used to appear at the end of the month in finance. Now it is a number the product team sees daily, next to quality and latency, and trades off deliberately.

Security and cost are the same conversation

Every cost control above also closes a security gap. Budgets bound abuse. Sending less reduces what leaves for the provider. Caching means fewer copies of sensitive content in flight. Routing small tasks to a model inside the client's boundary can remove an external transfer altogether. We present them together because they are decided together.

Purposefully cheap

The goal is not the lowest bill. It is a product whose cost is understood, predictable and proportional to the value it delivers, so that the business can let usage grow without fear. That takes a week of design and a dashboard, which is far cheaper than the surprised conversation in month two.

  1. Look at the cost distribution; find the tail.
  2. Send less: trim history, retrieve passages, precompute what repeats.
  3. Route by difficulty to the smallest adequate model.
  4. Cache exact requests, retrieval results and prompt prefixes.
  5. Set budgets per user, session and feature.
  6. Put cost next to quality in the evaluation gate.