Your RAG pipeline is retrieving the wrong chunks

A retrieval-augmented system that answers badly is almost never answering badly because the model is weak. It is answering badly because it was handed the wrong context and had no way to say so. The generation step is the visible part, so it takes the blame, but by the time a prompt reaches the model the mistake is already several steps old.

We have picked up enough of these systems mid-flight to recognise the shape of the problem. The team has tried three embedding models, moved to a bigger context window, and rewritten the system prompt twice. Retrieval quality was never measured, so none of those changes could have been evaluated.

Chunking is the real model

The default in most tutorials is a fixed window — 512 tokens, 50 tokens of overlap, split on whitespace. It is easy to reason about and it destroys the thing you are trying to retrieve. A policy document does not contain 512-token ideas. It contains clauses, and a clause that begins on one side of a boundary and resolves on the other is now two chunks, neither of which means what the clause meant.

The failure is quiet. Both halves still embed. Both still return plausible cosine similarities. The retrieved passage reads like an answer, and the model, having no signal that it is holding half a sentence, completes the thought on your behalf.

Splitting on document structure instead of token count costs an afternoon and usually moves accuracy more than a model upgrade will. Headings, list items, table rows, and clause boundaries are all better split points than the 512th token.

def chunk(doc: Document, max_tokens: int = 512) -> list[Chunk]:
    """Split on structure, then pack — never mid-clause."""
    chunks: list[Chunk] = []

    for section in doc.sections:
        buffer: list[Block] = []
        size = 0

        for block in section.blocks:
            tokens = count_tokens(block.text)

            # A single block over budget is split on sentences, not tokens.
            if tokens > max_tokens:
                flush(chunks, section, buffer)
                buffer, size = [], 0
                chunks.extend(split_sentences(section, block, max_tokens))
                continue

            if size + tokens > max_tokens:
                flush(chunks, section, buffer)
                buffer, size = [], 0

            buffer.append(block)
            size += tokens

        flush(chunks, section, buffer)

    return chunks

The chunk you embed is not the chunk you send

These are two different jobs and they want two different shapes. Embedding wants a tight, single-topic passage, because averaging several topics into one vector puts it somewhere between them and near nothing. The model wants enough surrounding material to interpret what it has been given.

So separate them. Embed the small passage, store a pointer to its neighbourhood, and expand at retrieval time. Similarity is computed on the precise thing; the model reads the passage in context.

• Embed the leaf passage — one idea, typically 200 to 400 tokens.
• Store the parent section id and the passage's position within it.
• On a hit, expand to the parent section, or to one passage either side.
• Deduplicate after expansion. Three adjacent hits expand into the same section and will otherwise be sent three times.

Similarity is not relevance

Cosine similarity measures whether two things are about the same subject. It does not measure whether one answers the other. A question about how to revoke an API key and a passage about how to issue one are highly similar and unhelpfully so.

This is what rerankers are for. A cross-encoder reads the query and the passage together and scores whether the passage answers the query — a slower computation, which is why it runs over the top fifty candidates rather than the whole corpus.

Retrieve fifty, rerank to five, generate. The extra hop costs about a tenth of a second and routinely does more for answer quality than anything downstream of it.

Measure retrieval on its own

The single most common gap in these projects is that nobody can say whether retrieval is working, because the only metric anyone looks at is whether the final answer seemed good. That number moves for at least four reasons and tells you which one it was.

Build a set of questions with the passages that should be retrieved for each, then track recall at k independently of generation. Two hundred labelled questions is enough to make the number stable, and a domain expert can produce that in a day.

With recall at k on a dashboard, the work stops being guesswork. Chunking strategies become comparable, the reranker's contribution becomes a number, and a model upgrade can be judged on whether it improved anything the retriever was already getting right.

Where to start on Monday

1. Write fifty questions and label the passages that answer them. Do this before changing any code.
2. Measure recall at 5 and at 20 against your current pipeline. This is the baseline every later claim is measured against.
3. Replace fixed-window chunking with structure-aware splitting, and prefix each chunk with its heading trail.
4. Add a cross-encoder reranker over the top fifty.
5. Re-measure. Only now is a different embedding model worth trying.

In most of the systems we have taken over, the first three steps close the majority of the gap, and they do it without touching the model at all.