RAG: The Librarian Who Reads Ahead

The librarian who reads ahead

Everything the agent has fetched so far, it fetched itself — it read the manual, searched the document, asked the library. Now meet the other arrangement. Imagine a librarian who sees your research question before you arrive, guesses what you'll need, and stacks those books on your desk. You never asked; the desk is simply loaded when you sit down. That is retrieval-augmented generation — RAG — and here is the whole machine, map first:

Walk the two halves. Ahead of time: cut the documents into chunks, compute each chunk's address in meaning-space, store them. At task time: compute the task's address, pull the chunks nearest to it, and paste their text into the prompt — before the model says a single word.

The model did not learn the documents. It was not trained,
not extended, not connected to a database.

We changed the DOCUMENT IT RESPONDS TO. Retrieval is prompt
construction — the one big prompt, built partly by machine.

And name who is holding the projector: whoever chunks, embeds, ranks, and inserts is casting shadows onto the agent's wall. RAG is a shadow-casting machine built out of code — which is exactly why its small design choices matter so much. Let's break it, one choice at a time.

"Use RAG" is not a method

Every arrow in the diagram hides a decision, and each decision can quietly change what the model believes. Watch four of them fail.

This is the tearing principle from earlier, now with a sharper consequence: at ingestion, a bad tear costs a strip its aboutness; at retrieval, it delivers half a rule as if it were whole. Chunking is a decision about which sentences are allowed to be separated — and rules travel with their exceptions or they lie.

That last line deserves its own alarm: retrieval always returns something. Nearest is not the same as near. The most dangerous RAG answer is the one assembled from confidently-inserted chunks that were merely the least-far-away.

How you hand over the stack

One more decision, and it may matter most of all: the framing of the retrieved text when it enters the prompt. Remember the no-wall principle — anything entering the conversation can instruct. The retrieved chunks enter at the user's seat. Watch the same chunk under two framings:

Retrieved text speaks from the user's seat, with the power
to instruct. Frame it as EVIDENCE, not ORDERS:

  "possibly relevant · retrieved automatically · ignore freely"

And remember what else rides in on retrieved text: anything
anyone ever wrote into the corpus. A stranger's words, delivered
to the seat that steers. Hold that thought — it returns when we
treat security.

The librarian and the stacks

So which is right — the librarian who reads ahead, or the agent who walks into the stacks itself, as it has all along? Put them side by side:

Neither wins outright, and production systems almost always mix them: pre-load the plausible; let the agent fetch the rest. The desk gets the obvious books; the stacks stay open, with the ladder's rungs — string search, meaning search, ask — available when the stack turns out to be wrong or thin.

Retrieval before the question is a BET.
Retrieval after the question is a PURCHASE.
Bets are cheap and sometimes wrong; purchases are exact and
cost iterations. Budget for both — and give the agent a way
to notice when the bet missed.