Retrieval-augmented generation, RAG, is the pattern behind most "chat with your documents" products: find the relevant passages, hand them to a language model, get an answer grounded in the source. Our first three RAG systems all demoed well and all had the same problem in production. They answered fluently from the wrong passage, or from the right passage of the wrong document version, and the fluency hid the error.

Here is what we changed, in the order we learned it.

Chunking is a product decision

The first system split documents into fixed-size pieces. It retrieved fragments that started mid-sentence and ended before the point, and the model filled the gaps with plausible invention. We moved to chunking by document structure: sections, clauses, table rows, with the heading path carried into each chunk so a paragraph about "termination" knows it belongs to "Section 8, Supplier obligations" and not to the customer's obligations two pages later.

For tables, which broke everything, we stopped treating them as text. Each row became a chunk with its column headers attached. For long procedures, each step kept its parent step.

A chunk should be something a careful human would be willing to quote on its own.

Retrieval needs more than similarity

Vector similarity finds passages that sound like the question. It does not know the question is about the 2026 policy and not the 2023 one, or that the user is in a region with a different rule. We added three things: metadata filters applied before ranking (document version, effective date, region, product), a keyword search alongside the vector search because exact terms like part numbers and clause references matter, and a re-ranking pass over the top candidates using a model that reads the question and the passage together.

The retrieval path we settled on: filter by metadata first, search two ways, re-rank, then generate with the passages attached.
The retrieval path we settled on: filter by metadata first, search two ways, re-rank, then generate with the passages attached.

Citations that a person can check

Early versions cited a document title. Users could not find the passage, so they stopped checking, so errors went unreported. Now every claim in an answer links to the exact passage, shown alongside, with the document version and date. Two consequences followed. Users caught errors quickly, which gave us the feedback we needed. And the model, prompted to answer only from the attached passages and to say when they did not cover the question, became noticeably more willing to say "the documents do not address this."

That refusal is a feature. A RAG system that always answers is a RAG system that sometimes invents.

Evaluation before launch, not after complaints

The change that mattered most was building an evaluation set before rewriting anything. We collected around two hundred real questions from the people who would use the system, and for each recorded the passage that a subject expert considered the right source. Then we measured two things separately: did retrieval return the right passage in the top results, and given the right passage, did the model answer correctly?

Separating them told us where the problem was. In two of the three systems, retrieval was the weak link and the model was fine. Tuning prompts, which is what everyone reaches for first, would have changed nothing.

Evaluating retrieval and generation separately. Most of our failures were retrieval failures wearing a fluent answer.
Evaluating retrieval and generation separately. Most of our failures were retrieval failures wearing a fluent answer.

When RAG is the wrong tool

Some questions are not retrieval questions. "How many open claims are over thirty days?" needs a database query, not a passage. "Summarise the trend" needs analytics. We now route: the system classifies the question and sends it to retrieval, to a structured query with the schema, or to a person, rather than forcing every question through the same path. Half of the "RAG is inaccurate" complaints we had received were questions RAG should never have been asked.

Security in retrieval systems

The index is a copy of your documents, so it needs the same protection as the originals. Two controls are non-negotiable. Permissions are enforced at query time: the retrieval step filters by what the current user is allowed to see, before the model sees anything, so a well-phrased question cannot surface a document the asker could not open. And document content is untrusted: a passage containing "ignore previous instructions" is data to be quoted, not a command to be followed, which means prompts are structured to delimit content and the system's behaviour is not controllable from the text it retrieves.

We also decide what leaves the environment. If the model is hosted by a provider, the passages sent to it are the client's documents. That is a data-transfer decision, with retention terms, and it belongs in the data map with everything else.

Purposeful RAG

Built carefully, RAG is the most useful AI feature most organisations can ship this year, because it makes what they already know reachable. Built carelessly, it is a confident liar with a search box.

  1. Chunk by structure, and carry the heading path with every chunk.
  2. Filter by metadata before you search; search by keyword and by vector; re-rank.
  3. Cite the passage, not the document, and show it.
  4. Build the evaluation set first; measure retrieval and generation separately.
  5. Route questions that are not retrieval questions elsewhere.
  6. Enforce permissions at query time and treat content as data.