---
title: "Why does the same question give different answers?"
description: "A model gives every possible next token a chance, and the app draws one at random by those chances, so two runs can differ. Temperature and top-p shape the draw."
topic: "AI systems"
chapter: "how-ai-answers"
order: 5
minutes: { plain: 6, deep: 14 }
prerequisites: ["Reading short Python functions", "Percentages and simple fractions", "What a token is"]
seo_title: "Why does ChatGPT give different answers each time?"
date: 2026-10-19
---
# Why does the same question give different answers?
Ask a chatbot one question twice and the replies can differ. This lesson uses the toy from [the last lesson](https://spicyshark.co/learn/why-answers-appear-word-by-word/), a tiny program that learned from 17 short recipe lines. Asked "How do I make tea?" 1,000 times, it wrote 16 different replies, and 12 of them were not among its own lines. Told always to take its likeliest token, it wrote one reply, 1,000 times. Real chatbots are far bigger, but the cause is the same: the model gives chances, and a separate step draws.
## In one minute
A chatbot gives different answers because, at each step, the model only gives every possible next token a chance, and the app then draws one at random by those chances, so two runs can take different paths. A token is a small piece of text, often a whole word. Picture a bag of 13 marbles, each marked with a token that followed the word "Wait" in the toy's lines. The model fills the bag; a hand reaching in makes the draw. Drawing by chance is called sampling. One different marble early means every later step starts from different text. Settings reshape the bag: temperature changes how many marbles each token gets, and top-p throws out the rarest.
## Explore the numbers
[Open the widget](https://spicyshark.co/learn/why-the-same-question-gives-different-answers/widget.html). The bars show the chances for the token after "Wait". Move Temperature to reshape them, move Top-p to cut the unlikely tail, and press Draw 20 times to see 20 picks from what is left. Under More settings, the seed, the starting number of the draws, makes them repeat.
## How it works
### The model gives chances and the app draws one
A pass is one run of the model ([Why do answers appear word by word?](https://spicyshark.co/learn/why-answers-appear-word-by-word/) covers the loop). Its output is a chance for every possible next token, and nothing more. In the toy, "Wait" was followed by three 4 times, ten 3 times, two 3 times, and four, one and six once each: 13 times in all. So three has 4 chances out of 13, which is 30.8%. This row counts what followed the single word Wait; inside a reply the toy looks back three tokens, so its chances there differ.
Choosing is a separate step. The app takes a random number from 0 up to 1 and sees which token's slice of the line it lands on. A token with a wider slice is hit more often. The pick joins the text, and the next pass starts.
*The number 0.5 is chosen for the picture; a real draw can land anywhere.*
### One early difference changes the rest
Each pick joins the text, and the next pass reads it. So one different draw changes every pass after it. Asked "How do I water a plant?", the toy's first and sixth replies shared their first 11 tokens (words and marks like the period each count as one), "Check the soil with a finger. If the soil is". Next came dry (3 chances in 4) or wet (1 in 4). One reply drew dry and wrote 24 tokens from dry on. The other drew wet and wrote 12 from wet on, and the two shared no whole sentence after that. Over 1,000 tea questions, the first token was Boil 797 times and Heat 203 times, close to its 80% and 20% chances.
### Temperature reshapes the chances
Temperature is a setting that reshapes the chances before the draw ([Why does AI make things up?](https://spicyshark.co/learn/why-ai-makes-things-up/) shows it in another toy). At 1, the chances stay as counted. Below 1 they sharpen: the likeliest token takes a larger share. Above 1 they flatten, and rare tokens gain. After "Wait", three goes from 30.8% at temperature 1 to 67.7% at 0.2, and down to 23.6% at 2. Twenty draws at 0.1 gave three 17 times; at 1 they gave three 7 times. As temperature nears 0, the likeliest token wins almost every draw. That is called greedy choice.
### Top-p cuts the unlikely tail
Top-p is a second setting, which cuts the tail before the draw. Rank the tokens from likeliest down. Keep the fewest whose chances add up to at least p, cut the rest, and scale what is left to total 100%. At p = 0.7, three, ten and two add up to 76.9% and are kept; four, one and six are cut and cannot be drawn. The kept chances become 40.0%, 30.0% and 30.0%.
This lesson applies temperature first, then top-p. Providers, the companies that sell chat models, may order them differently, and the pages read for this lesson do not say which order theirs use.
### A seed makes a run repeatable
A seed is the starting number of the generator that makes the random numbers. The same seed gives the same numbers, so the same draws. In the toy, seed 1 gives three 7 times, ten 4, two 3, four 2, one 1 and six 3 in 20 draws, every time. Seed 2 gives 5, 2, 6, 2, 3 and 2. Pressing Draw again with seed 1 goes on from where the numbers stopped, so it gives new picks.
Providers differ. Anthropic's reference page for sending messages lists no seed. OpenAI's reference for chat completions calls its seed a deprecated beta feature that tries to repeat a run but does not guarantee it. Google's Gemini reference has a seed and says the request uses a random one if none is set (all read 2026-10-19).
## What this means when you build
- **Do not expect the same answer twice.** Each reply is one draw. When you check an answer by hand, save the reply you checked instead of asking again.
- **Test with many runs.** A prompt that works once may fail on the next draw. Send it 20 times and count the outcomes, as the widget counts picks.
- **Lower temperature for steadier answers, not truer ones.** Low settings repeat more, but [a lower temperature does not make an answer true](https://spicyshark.co/learn/why-ai-makes-things-up/#why-a-lower-temperature-does-not-make-facts-true).
- **Check what your model accepts.** Some models reject or ignore these settings, so read your provider's parameter page before relying on them.
- **Never promise a repeat.** Anthropic's page says that even at 0.0 results can still differ from run to run. One 2025 test of a freely downloadable model got 80 different replies from 1,000 identical requests at temperature 0.
## Go deeper
*Optional. This part is for readers who write Python. The code is in the `code/` folder next to this lesson. `python3 show_numbers.py` prints every number in this lesson, and `PYTHONDONTWRITEBYTECODE=1 python3 -m pytest -q -p no:cacheprovider code/` runs the tests.*
### The sampler
`sampler.py` loads the toy from [the last lesson](https://spicyshark.co/learn/why-answers-appear-word-by-word/), so both lessons use the same 17 lines, tables and generator, and adds four small functions. (Docstrings are left out here.)
```python
def with_temperature(chances, temperature):
powered = [(token, chance ** (1 / temperature)) for token, chance in chances]
total = 0.0
for _, chance in powered:
total += chance
return [(token, chance / total) for token, chance in powered]
def top_p(chances, p):
if p >= 1:
return list(chances)
kept, running = [], 0.0
for token, chance in chances:
kept.append((token, chance))
running += chance
if running >= p - SLACK:
break
return [(token, chance / running) for token, chance in kept]
def choose(chances, u):
running = 0.0
for token, chance in chances:
running += chance
if u < running:
return token
return chances[-1][0]
def draw(chances, state, n=20):
picks = []
for _ in range(n):
state, u = toy.seed_step(state)
picks.append(choose(chances, u))
return picks, state
```
The model's part ends at `chances`. `choose` is the whole draw: walk down the list, adding chances, and stop at the first token whose running total passes `u`. The explicit loop for the total, instead of `sum`, makes Python and JavaScript add in the same order. `SLACK` is 1e-9: in floating point 0.7 + 0.2 is 0.8999999999999999, and without the slack a cut at exactly 0.9 would keep one token too many.
### Temperature is a division before softmax
[How softmax turns scores into shares](https://spicyshark.co/learn/online-softmax/) says a model's last step turns a score for each token into chances. If the chances are `p` and the scores `s`, then dividing every score by T before that step gives chances proportional to `exp(s/T)`, which is `p` raised to the power 1/T. So `with_temperature` can work on chances directly: raise each to 1/T and scale back to a total of 1. A test confirms it against a softmax of scores divided by T, within 1e-12. It refuses T of 0 or below, because it would divide by zero; greedy choice is a separate rule.
### One row by hand, at two temperatures
Take pass 11 of the tea reply, the text ending "bag in a", where cup followed 3 times and mug twice.
| temperature | cup | mug |
|---|---|---|
| 1 | 0.6 | 0.4 |
| 0.5 | 0.36 | 0.16 |
| 2 | 0.7746 | 0.6325 |
At temperature 1 the chances are 3/5 and 2/5. At 0.5 the power is 1/0.5 = 2: 0.6 squared is 0.36 and 0.4 squared is 0.16, which total 0.52, so cup is 0.36/0.52 = 69.2% and mug 30.8%. At 2 the power is 0.5, the square root: 0.7746 and 0.6325 total 1.4071, so cup is 55.1% and mug 44.9%. (The table shows the numbers before they are scaled to total 1.) Now draw. A random number of 0.5 picks cup at all three temperatures. A random number of 0.65 picks mug at 1 and at 2, but cup at 0.5, where cup's slice reaches 0.692. The same number can give a different token, because temperature moved the slices.
### Top-p, top-k and ties
Holtzman, Buys, Du, Forbes and Choi, "The Curious Case of Neural Text Degeneration" (2019 preprint, ICLR 2020, arXiv:1904.09751), report that choosing the likeliest text gives dull, oddly repetitive output, and propose nucleus sampling: drawing from a nucleus of the likeliest tokens, which leaves out the less reliable tail. Top-p is that method. Top-k keeps a fixed number of likeliest tokens instead of a fixed share of the chance; Anthropic's page describes its `top_k` as sampling only from the top K options for each next token (read 2026-10-19).
In the toy, ties at the cut go to the token that sorts first. At p = 0.8 the cut falls inside the three tokens with 7.7% each, so four is kept, since it sorts before one and six. The order matters too. With temperature 0.2 and p = 0.7, the order used here keeps three and ten (80.8% and 19.2%). Cutting first and sharpening after keeps three, ten and two (67.8%, 16.1%, 16.1%). Tests pin both.
### Two runs part ways
`first_split` finds the first position where two token lists differ, and `replies` writes many replies from one generator, as if the reader pressed regenerate again and again.
```python
def first_split(a, b):
for i, (x, y) in enumerate(zip(a, b)):
if x != y:
return i
return None if len(a) == len(b) else min(len(a), len(b))
```
With seed 1 at temperature 1, replies 1 and 6 to the plant question split at token 11.
The toy looks back only three tokens, so two replies can reach the same three tokens again. Their sentences still differed from the fork on, which is all this lesson claims.
### How many different replies
`python3 show_numbers.py` writes 1,000 tea replies from one seed at each setting. In the table, replies is how many different replies came out, and top is how many times the most common one came up.
| temperature | replies | top |
|---|---|---|
| 0.05 | 2 | 999 |
| 0.2 | 6 | 781 |
| 0.5 | 15 | 391 |
| 1 | 16 | 234 |
| 2 | 16 | 144 |
At temperature 1, 12 of the 16 are not among the 4 different replies in the toy's 5 tea lines, and top-p 0.7 cuts them to 4 different replies, 2 of them new. At 0.01 every reply is the greedy one: "Boil some water. Put a tea bag in a cup. Pour the water over the bag. Wait three minutes. Pull the bag out. Enjoy." This is a toy at one seed, so only the direction carries over: lower settings give fewer, more repeated replies.
### Does the draw follow the chances?
A frequency check draws 100,000 times from seed 1 and compares each token's share with its chance. The largest gap is 0.18 points at temperature 1, 0.16 at 0.5, and 0.11 at temperature 2 with p = 0.8. The test allows 1 point, and a fixed seed keeps it from being flaky.
### Why a real service still varies
Even if a service always takes the likeliest token, its answers can differ. Horace He and colleagues at Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference" (blog post, 10 September 2025, read 2026-10-19), ran 1,000 completions of one prompt at temperature 0 on Qwen3-235B-A22B-Instruct-2507, each 1,000 tokens long, and got 80 unique completions; the most common appeared 78 times. The first 102 tokens matched in all of them, and the 103rd differed: "Queens, New York" in 992 and "New York City" in 8. The authors name the main cause as server load: a request's result can depend on how many other requests are batched with it, and that number changes from moment to moment. With their batch-invariant kernels, which give the same result whatever the batch size, all 1,000 completions were identical. This is one lab's report on one model; it does not describe any particular chat service.
### What the toy leaves out
- It counts what followed what, with no neural network, so its chances come from 17 lines, not from a model's scores.
- It picks among a handful of followers at each step, where a real model gives a chance to every token in its vocabulary.
- Its random numbers come from a small whole-number generator, so Python and JavaScript agree. A service uses its own. The generator throws away its first four numbers, because a small seed's first numbers are far from random: on the tea question, Heat comes first in 0 of 1,000 seeds without that step and 201 with it, against a 20% chance. [The last lesson](https://spicyshark.co/learn/why-answers-appear-word-by-word/)'s seeded option does not do this, so the same seed draws differently there.
- It applies temperature, then top-p. Providers may use other orders, and may add other rules.
- It makes plain replies. Models that reason first may fix these settings: OpenAI's guide for GPT-6 says to remove `temperature` and `top_p` when reasoning effort is not `none` (read 2026-10-19).
### Proof it works
```bash
PYTHONDONTWRITEBYTECODE=1 python3 -m pytest -q -p no:cacheprovider code/
python3 code/embed_widget.py --check
```
The tests check these claims. The Wait row is three 4, ten 3, two 3 and one each for four, one and six, 13 in all, and the tea rows are [the last lesson](https://spicyshark.co/learn/why-answers-appear-word-by-word/)'s. Temperature 1 changes nothing, a low one sharpens, a high one flattens, the total stays 1 and the order stays, and the power rule equals softmax of scores over T. Top-p keeps three, ten and two at 0.7 and rescales them to 40, 30 and 30; ties go to the token that sorts first; rounding does not pull in an extra token; p of 1 keeps everything. Temperature comes before top-p. The same seed gives the same draws and the next press goes on from the last; 20 draws at seed 1 give 7, 4, 3, 2, 1 and 3. A frequency check agrees with the chances to within 1 point. At a very low temperature the reply is the greedy one. Replies 1 and 6 to the plant question split at token 11, dry against wet.
`widget.html` runs the same rules in JavaScript, so `test_widget.py` runs the widget's own code in Node and compares it with the Python: the generator, every chance the sliders can reach (four rows, 40 temperatures and 19 top-p values), the cut, and 1,000 draws for 72 combinations of row, setting and seed. A second test drives the page with a stand-in document, and checks the opening settings, the bars, the counts, the note for each press, a change of row on its own, Start over, the limits on each control and the theme message. `embed_widget.py --check` fails if the data inside the widget differ from the toy.
### Where this is used in practice
Provider documentation (all read 2026-10-19) shows how far the settings go. Anthropic's Messages API page lists `temperature`, `top_p` and `top_k` as deprecated: models released after Opus 4.6 reject any `temperature` other than the default 1.0 and any `top_p` below 0.99. OpenAI's Responses reference gives a range of 0 to 2 for `temperature` and recommends altering one of `temperature` and `top_p`, not both. Its guide on reproducible outputs says Chat Completions are non-deterministic by default, and that the same `seed` and parameters give mostly the same output, with `system_fingerprint` showing backend changes. Google's Gemini API reference lists `temperature` from 0.0 to 2.0, `topP`, `topK` and `seed`, and says the first three are ignored on Gemini 3.6 Flash and newer models. None of these pages promises identical output on every run.
## Skills this lesson builds
- Explain why the same question gives different answers: the model gives chances, a separate step draws, and one early difference carries on.
- Predict what temperature and top-p do to a row of chances, and work one row by hand.
- Use a seed to repeat a draw in a toy, and say what a provider's seed does and does not promise.
- Measure variety by counting outcomes over many runs, not by trusting one.
- Read a provider's parameter page for what a model accepts, and what it ignores.
**Portfolio project:** write a script that sends one prompt to any chat model 20 times at each of two temperature settings (or at the default, if the provider fixes it), and prints a table of how many different replies came back and how often the most common one appeared. Publish the prompt, the script, the table and the parameters the provider accepted.
## Check yourself
1. Why can two runs of the same question give different answers?
Answer
At each step the model gives every possible next token a chance, and the app draws one at random by those chances. A different draw early changes the text every later step starts from.Answer
The chances flatten: after "Wait", three falls from 30.8% to 23.6% and the rarest tokens rise from 7.7% to 11.8%. More different tokens get drawn, so the answers vary more.Answer
The first two, because 40% + 30% reaches 70%. They are scaled to total 100%: 40/70 is 57.1% and 30/70 is 42.9%. The other two are cut and cannot be drawn.Answer
Cup. Its slice now runs from 0 to 0.692, and 0.65 falls inside it. Temperature moved the slices, so the same random number gives another token.Answer
No. Anthropic's page says that even at 0.0 results can still differ from run to run, and one 2025 test found 80 different replies from 1,000 identical requests at temperature 0 on one downloadable model. Count outcomes over many runs and plan for a different one.