Stop 5 · Why it makes things up
A fluent wrong answer and a fluent right one are produced by the same process. Which is exactly what tools are for.
Every model behaviour on this stop is a written illustration, not a live API call. The numbers are computed from the assumptions stated below.
What time is it?
Why the answer cannot be inside the modelThe answer changes continuously, and the model is the result of one training run: nothing that happened afterwards is in it.
The answerIt is around three in the afternoon.
✕ Can you check itNothing to check against: there is no source it could show you. The sentence sounds exactly like a correct one.
Which test framework does this project use?
Why the answer cannot be inside the modelThe answer is in your repository, and your repository was not in the training data. It knows what a test framework is; it does not know which one you picked.
The answerIt looks like Jest; projects of this shape usually use it.
✕ Can you check itThe only thing behind it is usually. It is describing what projects of this shape tend to do rather than what yours does — and the two read identically.
How is formatMoney implemented in lib/cart.ts?
Why the answer cannot be inside the modelThat function does not exist. The question presupposes that it does, and continuing a passage of text is the thing the model is best at.
The answerIt formats a number into a currency string with Intl.NumberFormat, in about ten lines.
✕ Can you check itA reasonable description, and entirely invented. Note that it did not hedge: it produced this sentence by the same process it would have used for a true one.
Three questions first
What the three questions above have in common is that none of their answers could be inside the model. The first changes by the minute, the second lives in your repository, and the third does not exist. Switch to without tools and all three replies are fluent, specific and confident, and none of them is reliable.
Look at the third one in particular. The question presupposes that formatMoney exists, and the model continues from that presupposition into a perfectly plausible description. It did not hedge, and it was not lying. It was doing the thing it always does: continuing a passage of text.
Why it cannot tell that it is inventing
Back to the sentence from stop 1: the model's entire job is to take text and produce more text. What decides the next is how well it fits after this context, not whether the resulting sentence is true. Fitting well and being true are strongly correlated in training and are not the same thing — and on anything it has not seen, they come apart.
So the invented sentence was produced by the same process as a true one. There are signals inside a model that track whether it knows something, but they are unreliable, and nothing makes them stop an invented sentence on its way out. Which explains a common disappointment: asking are you sure usually achieves little. What comes back is mostly a well-fitting passage about certainty, not a second look — it has nothing to look at except what it just wrote.
Beginner question: Can you not just tell it to say when it does not know?
It helps, within limits, and the reason is worth understanding: the instruction raises how willing it is to say I do not know, but it cannot make the model reliably tell what it knows from what it does not. So it reduces invention without removing it; sometimes it also makes the model decline questions it would have answered correctly, while the wrong answers that felt familiar still come through. Pair the sentence with a place to look and it works much better.
This is what tools are for
stop 1 said tools exist so the model has hands: to read a file, to run a command. Here is the other half, and the more fundamental one: tools exist so it has somewhere to look. A model asked what time it is has to guess. A model with a clock does not. The difference is not that it became smarter; it is that the answer moved from an impression about text to a tool_result in the .
The third question makes this clearest. The most underrated thing a tool does is not to supply a fact but to supply the absence of one: search_files("formatMoney") returning nothing is itself evidence that can be written into the array, and that is what gives the model grounds to say the function does not exist. Without a tool, "I do not know" has no shape, because nothing is pushing it in that direction.
A citation can be invented too
The usual next move is to demand sources: retrieve the material into the array and have the answer cite it. That works — it is usually called retrieval-augmented generation, or RAG — and it is the most practical technique available. It has one boundary worth being clear about: a citation is also generated text. A filename it never read, or a line number assembled to look right, is indistinguishable from a real source.
So the technique that pays is not requiring a citation but requiring a citation you can check, and then checking it. A reference to lib/client.ts line 4 is worth something because opening the file falsifies it in seconds. A reference to the project documentation is worth nothing, because it points at nothing you can open. A tool that returns a filename and a line number makes verification a few seconds' work — how to say in a tool's description what it returns is covered at stop 7.
Beginner question: Does retrieving the material stop it inventing?
Much less, not none, and there are three gaps worth remembering. When retrieval finds nothing relevant the model still answers, unless your tool says plainly that nothing came back. What was retrieved can itself be wrong or out of date, and the model is in no position to doubt it. And retrieved material is outside content, which may be talking to the model — that is stop 10. Retrieval replaces inventing from impressions with depending on what you supplied, which makes the quality of what you supplied the new ceiling.
The model is continuing a passage, not consulting a record, so "I do not know" has no natural shape for it. A tool gives it that shape: I looked, and nothing came back, therefore there is none. Do not ask whether it knows. Give it somewhere to look.
A look back: what the model is like
Three questions. The first two are two faces of one misconception: that a sentence can make a model reliable.
Nothing is scored here. Every question can be answered again, and a wrong answer names the idea it came from.