"Should we fine-tune a model on our data?" is one of the most common questions we hear, and the answer is usually no, not yet. Not because fine-tuning does not work, but because two cheaper techniques solve most of the problems people bring to it. This post sets out how we decide among prompting, retrieval and fine-tuning, and describes one project that needed each.

Three techniques, in order of cost

Prompting changes the instructions: what the model is told to do, the format it must produce, the examples it is shown. It costs a few hours, changes in minutes, and works when the model already has the capability and just needs to be told the task.

Retrieval changes what the model knows at the moment of the request: it finds the relevant facts, documents or records and attaches them. It costs days to weeks, keeps knowledge current without retraining, and works when the problem is that the model does not have your information.

Fine-tuning changes the model's behaviour: its style, its handling of a narrow domain, its reliability on a specific format. It costs weeks, needs hundreds to thousands of good examples, must be repeated when the base model changes, and works when the problem is a behaviour that instructions and examples cannot reliably produce.

The decision as we run it: instructions first, then knowledge, then behaviour. Each step is tried and measured before the next.
The decision as we run it: instructions first, then knowledge, then behaviour. Each step is tried and measured before the next.

The project that asked for fine-tuning

A client wanted a model fine-tuned to write product descriptions in their brand voice, because the generic output "sounded wrong." We asked to see the prompt. It said, in full, "write a product description for this item." We spent a day on it: a description of the voice with do-and-don't examples, six real descriptions the client loved, the product attributes structured rather than pasted, and a format specification. The output went from sounding wrong to needing light edits. No fine-tuning, no training data, no model to maintain.

The cheapest experiment is a better prompt, and it has to fail before anything else is justified.

The project that needed retrieval

A support assistant for a software product answered plausibly and often wrongly, because the model's general knowledge about the product was out of date and incomplete. Prompting could not fix that; the model did not know the current behaviour. Retrieval did: the current documentation, release notes and known issues, indexed and attached to each question with citations. The knowledge stayed current because the index was rebuilt from the documentation on every release, which no fine-tuned model could have matched.

The project where fine-tuning was the answer

A document-extraction pipeline had to read a specialised class of forms and produce a strict structured output. Prompting with examples reached a plateau: the model handled common layouts but was inconsistent on the long tail, and each prompt tweak fixed one case and broke another. Retrieval did not apply; the knowledge was in the form, not in a library. What we had was three thousand reviewed extractions from the review queue, each a correct example. Fine-tuning a smaller model on those made it more consistent than the large model with prompting, cheaper per document, and fast enough to run inside the client's own environment, which their data classification required.

The signals that fine-tuning was right: a narrow, repetitive task; a plateau under prompting; a large set of reviewed examples that already existed; and a reason to prefer a small model, which was cost and data residency.

What each technique changes, and the signal that tells you the previous one has run out.
What each technique changes, and the signal that tells you the previous one has run out.

The measurement that makes the decision honest

Every one of these choices was made against an evaluation set, not an impression. The prompt change was measured against fifty descriptions the client had rated. The retrieval system was measured on whether the right document was found and whether the answer matched it. The fine-tuned extractor was measured on field accuracy against held-out reviewed forms. Without that, "fine-tuning made it better" is a feeling, and feelings are how teams end up maintaining a model they did not need.

Security and cost differences

The three techniques have different security shapes. Prompting sends your instructions and examples to the provider on every request. Retrieval sends the retrieved content, which may be sensitive, so it needs permission filtering and a decision about what leaves the boundary. Fine-tuning sends your training data once to wherever the training runs, and then the model itself becomes an asset that contains your data in some form and must be stored and served securely. If the training data includes personal information, fine-tuning creates a new copy of it that is hard to delete on request; this is a reason on its own to prefer retrieval, where deletion is removing a document from an index.

On cost, prompting is cheapest to build and the most expensive per request if prompts are long. Retrieval adds infrastructure but keeps prompts focused. Fine-tuning is expensive up front and can be the cheapest per request when it lets a small model replace a large one.

Purposeful choices

Start with the prompt, measure, and move on only when the measurement says so. Retrieval when the model lacks your knowledge; fine-tuning when it lacks a behaviour that examples cannot teach and you already have the data to teach it. The order is not a rule of thumb. It is the order of cost, and cost includes the model you will have to maintain for years.

  1. Write the best prompt you can and measure it.
  2. If the model lacks your knowledge, add retrieval.
  3. If a narrow behaviour plateaus and you have reviewed examples, consider fine-tuning.
  4. Decide every step on an evaluation set.
  5. Weigh the security shape: what leaves, what is copied, what can be deleted.