RAG is an open-book exam for AI

At the end of part one I left you with a slightly bleak picture. The thing you’re talking to isn’t searching anything. It’s a file of numbers that got very good at guessing what text comes next, and that file was finished on a particular day and hasn’t changed since.

That leaves an obvious hole in the story. You’ve uploaded a PDF to one of these tools and had it answer questions about a document that didn’t exist when the model was made. You’ve asked about this morning’s news and got an answer with links underneath. A file of frozen numbers shouldn’t be able to do either of those, and on its own it can’t.

Here’s the whole answer in one sentence, and then I’ll take it apart.

Before the model answers, a separate piece of software finds the pages most likely to contain the answer and puts them in front of it.

It goes by an ugly name, retrieval-augmented generation, or RAG, and even the person who named it isn’t fond of it. Patrick Lewis led the 2020 research paper that coined the term, and in an interview he happened to give from Singapore, he told NVIDIA’s blog, “We definitely would have put more thought into the name had we known our work would become so widespread.”

I build products on top of these models for a living, and wiring one up to a pile of documents is about the most common job in that line of work. It’s also the part people most often assume is magic, when it’s much closer to a filing system.

A model answering from memory is sitting a closed-book exam

Think back to school. In a closed-book exam you answer from whatever stuck, and when nothing stuck you write something that sounds right and hope. That’s a model on its own, and it’s why its wrong answers arrive with the same confidence as its right ones.

An open-book exam is a different sport. You’re allowed to bring the textbook, and the skill being tested shifts from remembering to finding the right page and reading it properly. You don’t know any more than the student beside you, you’ve just got the page open.

The model didn’t learn your document. Somebody found the right page and put it on the desk a second before it answered.

Drawn out, it’s three steps.

The model never sees your whole library

The first surprise is how little of your material the model actually gets. Connect one of these tools to a company’s shared drive, or to the whole internet, and it can’t read all of that every time you ask something. A model can only hold so much text in front of it at once, every word it reads costs time and money, and the more you pile on, the more likely it is to miss the one line that mattered.

So the documents get prepared in advance. They’re cut into short passages, a few paragraphs each, and filed in a way that makes them quick to search. When you ask a question, only a handful of those passages get handed over. For one short PDF, some apps skip all this and pass along the whole thing.

The librarian matches meaning, not spelling

The fetching step is the clever bit, so slow down here.

An ordinary search looks for the words you typed. Ask a staff handbook “can I carry my leave over to next year” and a keyword search finds nothing, because the paragraph you need is headed “Forfeiture of unused annual leave” and shares almost no words with your question.

The fetching step works on what a passage is about. Every passage gets turned into a long list of numbers that acts like coordinates on a map of meaning, and passages about similar things end up near each other on that map. Your question gets coordinates too, and the system picks up whatever is sitting closest. The leave question lands right beside the forfeiture paragraph, even though the two barely share a word. Nobody wrote a rule saying those belong together, which is the same trick part one was about, numbers standing in for meaning. The trade name for the coordinates is embeddings, which you’re welcome to forget.

Then it goes back to guessing, with the page open

Here’s what the model actually receives. Above your question, where you can’t see it, the system has pasted the passages it fetched, with an instruction along the lines of “answer using the material below”. Then the model does what it always does and works out what text would most plausibly come next. What’s changed is what counts as plausible, because with the relevant paragraph sitting right there, the likeliest continuation is one that repeats what the paragraph says.

And because the system knows which passages it handed over, it can show you small numbered footnotes, which is where the links under an answer come from.

This is how nearly everything you’d describe as “AI that knows about my stuff” works. Uploading a PDF, Google’s NotebookLM, a bank’s help chatbot and every assistant that “searches the web” are the same three steps with a different pile of documents. When ChatGPT or Gemini looks something up online, a search engine does the searching and the model does the reading. Google said as much about its own AI Overviews, when Liz Reid, the vice president who runs Google Search, wrote that “They’re not simply generating an output based on training data.”

An open book can still be the wrong book

That post from Google exists because of a failure. In May 2024 its AI Overviews told people to add glue to pizza sauce so the cheese would stick, and the source turned out to be an old joke on a forum that the fetching step had picked up as if it were advice. Reid put it down to “sarcastic or troll-y content from discussion forums”. The model did its job faithfully, on a page nobody should have handed it.

These systems go wrong in three ways, and only the last is the model’s own doing.

The wrong pages get fetched. The answer is in your documents, but the passage that contains it never gets handed over, so the model answers from the near-misses it was given, or quietly from memory, and it won’t tell you which.

The pages are wrong. The passage might be an out-of-date policy, a forum joke, or something written on purpose to be picked up. I’ve written about people already gaming this with thirteen words planted in a Reddit comment.

The page gets misread. A footnote shows that a passage was handed over. It doesn’t show that the passage says what the answer claims.

What to do with this

When the answer matters, don’t ask from memory. Upload the policy, the contract or the manual, and ask your question about that, which turns a closed-book exam into an open-book one for the price of ten seconds.

Then check the book. Click one footnote and read the passage it points to, and if the answer says something you can’t find there, ask which part of the document it’s relying on. A vague reply usually means the right page was never fetched.

This is the second piece in a series I’m writing to teach this properly, from the bottom up, without the maths. The next one is about what happens to the things you hand over, and what should never go into the box at all. It has to come third, because I’ve just told you to upload your documents, and that advice shouldn’t travel without the warning.

If you want the longer version of this one, NVIDIA’s explainer is where that line about the name comes from. Fair warning, it gets technical about halfway down.