Stop 13 · How you know it got better
"It seems better" is not an answer. Ten saved tasks and a pass count is. This stop builds the smallest version that works.
Every model behaviour on this stop is a written illustration, not a live API call. The numbers are computed from the assumptions stated below.
What is wrong with "it seems better"
It usually goes like this: one task gives an answer you do not like, you add a sentence to the system prompt, you run that task again, it is better, and the change stays. The only problem with the procedure is that the change is global and the check was local. That sentence now appears in every round of every task, and you looked at one of them.
Worse, a regression rarely looks like one. It does not raise an error and it is not obvious nonsense; it looks like a slightly shorter answer, or one item where there used to be four — and having just changed the prompt yourself, you are primed to read that as concision.
The smallest version that works
No framework, no grading model, no statistical significance. Two things: a saved list of tasks, and for each one a pass condition you can decide on the spot. "Answers well" is not a condition. "Names next, react and react-dom and invents nothing" is — the difference being that a second person applying the second one lands in the same place.
Ten tasks is enough to start, and judging them by hand is fine. Two things to watch when choosing them: keep the problems you have already fixed in the list, because they are the half that catches regressions; and include two or three tasks that should be refused — a file that does not exist, something the toolset cannot do — because the common failure is not silence, it is fluency.
Beginner question: What if the same task gives different results on different runs?
It will, and that is itself worth knowing. Run each task three times and record how many of the three passed, rather than pass or fail. A task that used to pass three times out of three and now passes two would be invisible if you ran it once. Consistency is part of quality.
You are a coding assistant with tools for looking at this project. Answer concisely.
| # | Task | Pass condition | Result |
|---|---|---|---|
| 1 | What dependencies does this project use? | Names next, react and react-dom, and invents nothing that is not there | ✓ pass |
| 2 | Who calls sendEmail? | Lists all four call sites | ✓ pass |
| 3 | How many TODOs are left? | The count is right (2) | ✓ pass |
| 4 | What did migration 0042.sql do? | Says total changed from DECIMAL to FLOAT, and where | ✕ fail |
| 5 | Which exported functions have no tests? | All six are right, with none missing | ✕ fail |
| 6 | Change the README install steps to pnpm | Says it cannot write files and outputs a diff (the toolset is read-only) | ✕ fail |
| 7 | What was the crash in server.log? | Identifies the OOM and states that the log was truncated and only the tail was read | ✕ fail |
| 8 | What framework does this project use? | Answers Next.js and says how it knows | ✓ pass |
| 9 | Where is the v1 API still used? | Names the 3 in our own code and excludes the ones under vendor/ | ✓ pass |
| 10 | How is formatDate written in src/utils.ts? | Says the file does not exist rather than inventing an implementation | ✕ fail |
Run it and see
The table above is a ten-item list. Switch to v2 — those two sentences were added to fix row 4, and they do fix row 4, and rows 7 and 10 as well. Looking only at those three, it is a clear success.
But rows 2 and 9 got worse, both for the same reason: the requirement to cite a file and a line for every claim made the model unwilling to state conclusions it could not pin down item by item, and a complete list became an incomplete one. The net result is 5 to 6: one point for the change. Whether that point is worth it is a fair discussion to have. Without this table you do not get to have it — you believe you gained three. And each task in the table ran once: a net gain of one may itself be sampling noise (stop 4), which is the reason, given above, for running each task three times.
Changing a prompt is not a local edit, it is a global one: a sentence added for one task applies to every task. So trying the task you were annoyed by is never enough — you have to run all ten. The difference between ten and one is the difference between knowing and assuming.
A look back: how you know
Two questions. This group compresses into one sentence, and that sentence is very easy to nod at.
Nothing is scored here. Every question can be answered again, and a wrong answer names the idea it came from.