Stop 4 · Why the same question gives different answers
The same task and the same array can take two different paths. This stop explains why, and what follows from it.
Every model behaviour on this stop is a written illustration, not a live API call. The numbers are computed from the assumptions stated below.
Task: how many TODO comments are left in this project?
At 0 the likeliest path is taken every time. The higher it goes, the more the others get a turn.
Press it a few times first
Above is one task: count the TODO comments left in this project. The is identical, the system prompt is identical, the tools are identical. Press Run it again a few times and watch what it decides to do.
Most of the time it searches straight away, sometimes it lists the directory first, sometimes it guesses a path that does not exist, and occasionally it just states a number without looking. Two of those four reach the right answer, one costs an extra round, and one produces a figure with nothing behind it — and you changed nothing.
The model rolls a die at every step
A model does not think of the next word. What it computes is a table: for every possible that could come next, how likely it is. Something then has to pick one out of that table. That step is called . It is not the network itself but a step the inference service runs after it; through an API it happens on the server, and your code can only influence it through parameters.
is the knob on that step. It does not change the table the model produced; it changes how the pick is made. Low temperature and the most likely token is chosen nearly every time; high temperature and the ones further down get a turn. Drag the temperature to 0 above and press again a few times: the same path every time. Drag it to 1 and the path that answers without looking starts to appear. (The slider shows the principle. Some models' APIs no longer expose temperature at all, and then the variation can only be narrowed through the prompt and the tool results.)
The part that matters is that this choice happens once per token generated. A reply is hundreds of tokens long, and one different pick early on carries everything after it somewhere else: choose "src" rather than "." as the third token and the whole run becomes a different run.
Beginner question: Does temperature zero make it fully deterministic?
Closer than you would think, and not quite. First, even with deterministic sampling the arithmetic on the server is not guaranteed to be bit-identical across batches and hardware, and two probabilities that sit very close together can swap. Second, and more importantly: an agent's input is not only your prompt, it is also whatever the tools returned. A file changed, a command was slower, a page was edited — the array is different, so the rest is different. Temperature zero removes the randomness in one place; it does not remove the fact that the agent runs in a world that moves.
So an agent is not a function
This is the thing to carry away. A function in the mathematical sense — a pure function, in programming — has the property that the same input gives the same output: test it once and you know how it behaves. An agent does not have that property. The same input can take a different path, which makes "it worked when I tried it" not evidence but one sample.
That sentence decides how the later stops work. Why a prompt change is checked against a whole set of tasks rather than the one that annoyed you (stop 13): because one pass says very little. Why the same task is run three times and recorded as how many of three passed: because a pass rate is a comparable quantity and a single result is not. Why production shows behaviour you never saw: because what you saw were the likeliest paths, and your users are sampling the rest on your behalf.
Beginner question: How do you debug something that is not deterministic?
Move your attention from why this run was wrong to how often this kind of run is wrong. In practice: run it several times and record the distribution rather than trying to reproduce one case; save the full array from each round, so that at least that path can be walked again afterwards; and keep the uncertainty away from the edges — approval for anything with a side effect (stop 11), and tools that are safe to call twice. You cannot remove the randomness. You can make being wrong cheap.
An agent is not a function. The same input can take a different path, so "it worked when I tried it" is not evidence — it is one sample. Turning the temperature to zero reduces the variation without removing it, because the environment differs between runs as well.