Evaluating an LLM application before launch, not after the complaints

The hardest question on any language-model project is not "does it work?" but "how would we know?" Traditional software has tests that pass or fail. A model's output is a paragraph, and whether it is good depends on who is reading. Early on we handled this the way most teams do: someone tried a few questions, it looked fine, we shipped. The complaints arrived with the first real users.

This is the evaluation practice that replaced "looks fine," and the release gate it became.

Start by collecting the questions people actually ask

Our evaluation sets used to be written by us. They tested what we thought the system should do and were blind to what users would do. Now the first step on any project is to gather real inputs: support tickets, the questions a team emails each other, the documents actually submitted last quarter. A hundred real examples beat a thousand invented ones, because real examples contain the ambiguity, the typos and the edge cases that break systems.

For each example we record what a good output looks like, written or approved by someone who does the job. That reference is the expensive part and it is the whole value of the exercise.

Measure the parts separately

An LLM application is a pipeline: retrieval, prompting, generation, post-processing, sometimes tool calls. If we only score the final answer, a failure could be anywhere. So we score stages. For a retrieval system: was the right source found? For an extraction system: was each field correct? For a classifier: the usual precision and recall. Only then the end-to-end quality.

On a document-extraction project this saved weeks. End-to-end accuracy was disappointing, and the instinct was to rewrite the prompt. Stage scoring showed the model was fine on the text it received; the failures were in the document parsing that fed it, which was dropping table cells. We fixed the parser and the prompt stayed as it was.

Score the stages. The final answer tells you something is wrong; the stages tell you what.

Where a model can be the judge, and where it cannot

Using a second language model to grade outputs is fast and useful within limits. It works well for checkable properties: is the answer grounded in the provided passage, does it follow the required format, does it contain the required fields, is it in the right language. It is unreliable for the judgements that matter most: is this the right answer, is this advice safe, would an expert agree. For those, a person has to grade, at least on a sample.

Our rule: model-graded checks run on every change and gate releases on the checkable properties. Human grading runs on a fixed sample before launch and periodically after, and its results are compared with the model grader to see whether the model grader can be trusted on that class of question.

The adversarial half of the set

Half of a good evaluation set is about correctness. The other half is about behaviour under pressure: prompt injection attempts, requests for data outside the user's permissions, questions the documents do not answer, hostile or confused inputs, and requests the system should refuse. Each has an expected behaviour, and "refuse clearly" is a passing answer for many of them. A system that scores well on correctness and has never been tested adversarially is a system whose first security test will be run by a stranger.

The release gate

Everything above turns into a gate. A change to a prompt, a model version, a retrieval setting or a parser runs the full set. Checkable properties must not regress. Human-graded quality on the sample must meet the threshold agreed with the business. Adversarial cases must all pass. Cost and latency per request must stay within budget. If any of these fail, the change does not ship, however good it looked in a demo.

The gate also runs when nothing has changed, on a schedule, because model providers update their models and the outputs shift underneath a fixed prompt.

Using evaluation purposefully

The evaluation set is the product specification in executable form. It tells the team what good looks like, catches regressions that a demo would hide, and gives the business a number to hold the system to. Teams that build it first ship with confidence. Teams that skip it ship with hope.

1. Collect real inputs; have experts write the references.
2. Score each stage, then end to end.
3. Use a model grader for checkable properties; use people for judgement, on a sample.
4. Keep an adversarial half, with refusal as a valid pass.
5. Gate every change and every provider update on the full set.