---
title: "Why does AI make things up?"
description: "A language model writes the likely next words, not checked facts. Where its training text says little, a fluent answer can be wrong, and it rarely says so."
topic: "AI systems"
chapter: "trust-the-answers"
order: 12
minutes: { plain: 6, deep: 11 }
prerequisites: ["Reading short Python functions", "Percentages and simple fractions"]
seo_title: "Why do AI chatbots make up facts?"
date: 2026-10-19
---

# Why does AI make things up?

Ask a chatbot for a study on sleep and memory, and it may give you authors, a journal and a year, in the right format, for a study that does not exist. It is not lying: it wrote the kind of text that usually follows such a question. This lesson uses a toy, a tiny program that learned from 37 short sentences, 30 of them like "The capital of France is Paris." Nothing it read says the capital of Peru. Asked to finish "The capital of Peru is", it picks Paris, which has the biggest share of any word, 10.5%. The true answer, Lima, gets 0.0%. Real models are far bigger; the lesson says where the toy misleads.

## In one minute

AI chatbots make things up because they are built to write the most likely next words, and the most likely words are not always the true ones. Picture a person who has read a huge pile of text and finishes your sentences the way such sentences usually go. For a famous fact, the usual ending is the right one. For a rare name, an exact number or a recent event, the pile says little, but a smooth ending is still easy to write. Nothing in the basic design checks the fact, so a made-up answer sounds as sure as a true one.

## Explore the numbers

[Open the widget](https://spicyshark.co/learn/why-ai-makes-things-up/widget.html). It is the toy from the opening, starting on Peru. Move the slider to add copies of the true sentence, "The capital of Peru is Lima.": one copy flips the answer to Lima, at 51.2%. Tick the box that allows "I don't know" when the top word is below 50%: with no copy added the toy declines to answer, and with one copy it answers Lima. Then move the temperature slider and watch Paris grow while Lima stays at 0.0%.

## How it works

### A language model writes the likely next words

A language model is a program that learned, from a very large amount of text (its training text), which words tend to follow which. To answer, it picks a likely next word, adds it to the text, and repeats, one word at a time. Real models work in tokens, small chunks of text ([What is a token?](https://spicyshark.co/learn/what-is-a-token/) explains them), but the loop is the same. Nothing in it looks a fact up or compares the sentence with a source. IBM's explainer (read 2026-10-19) says a generative model aims at what fits the pattern in its training data, not at what is true.

![Boxes joined by arrows: question, pick a likely next word, repeat, fluent answer. A dashed box, check against a source, joins nothing and is labelled not built in.](https://spicyshark.co/learn/why-ai-makes-things-up/figs/no-lookup-step.svg)

### Where the text says little, a fluent answer can be wrong

Some facts show up many times in text. The toy read "The capital of France is Paris." five times, and asked about France it answers Paris with an 85.1% chance. Other facts show up once or never. The toy never read Peru's capital, yet it must choose a word after "is", and in what it read, those are almost all capitals. So it chooses the capital it saw most often, Paris, and gives Lima 0.0%. The answer is fluent because it fits the pattern, and wrong because the pattern is not the fact.

![Bars for the toy's chance of each word after The capital of Peru is: Paris 10.5%, Rome and Tokyo 6.5%, Berlin and Cairo 4.5%. The true word, Lima, has no bar: 0.0%.](https://spicyshark.co/learn/why-ai-makes-things-up/figs/peru-next-word.svg)

Real models are not this blunt: they squeeze their training text into numbers, and can often answer a question worded in a way they never saw. Still, rare facts are a weak spot. Kandpal and colleagues (2022; abstract read 2026-10-19) report that a model's ability to answer a fact question relates to how many documents about it appeared in training. Kalai and colleagues (2025; read 2026-10-19) argue that for facts with no pattern to learn, such as a birthday, a model fresh from its training text should get wrong at least the share of them seen exactly once, provided its confidence is calibrated (it matches how often the model is right).

### Why tests push towards a guess

Builders compare and choose models by their scores on tests. Most tests give 1 point for a right answer and 0 for a wrong one, and also 0 for "I don't know". So a guess can only gain points, and a model that always guesses scores higher than one that sometimes says "I don't know". The 2025 paper argues that hallucinations (answers that sound right but are false) persist because most tests are graded this way. The toy has no test score: this cause is the paper's argument, not something the toy shows.

![Points on a right-or-wrong test: answer and be right, 1 point; answer and be wrong, 0; say I don't know, 0. A wrong guess costs no more than a blank.](https://spicyshark.co/learn/why-ai-makes-things-up/figs/one-or-zero.svg)

### Why a lower temperature does not make facts true

Many chat tools have a setting called temperature that controls how adventurous the word choice is: at a low setting the model almost always takes its top choice, and at a high one it takes more varied words. Lowering it makes answers more repeatable, not truer: the top word is taken whether or not it is true. In the toy, going from the counted chances (setting 1) to the lowest setting lifts Paris from 10.5% to 98.3%, and Lima stays at 0.0%.

![Bars for the toy's chance of Paris at seven temperature settings from 0.1 to 2, including 98.3% at 0.1, 57.4% at 0.3, 10.5% at 1 and 4.2% at 2. Lima is 0.0% at every setting.](https://spicyshark.co/learn/why-ai-makes-things-up/figs/temperature-bars.svg)

## What this means when you build

- **Give the model the text and ask it to quote.** If the answer should come from a document, put the document in the prompt (the text you send the model) and tell the model to use only that document. For a long document (Anthropic's guide, read 2026-10-19, gives about 20,000 tokens as "long"), ask it to pull out the relevant quotes first. Your app can then check each quote with a plain text search.
- **Ask for sources, then check them.** The same guide describes asking the model to cite a quote for each claim and to withdraw any claim it cannot back. Open and read any link, paper or court case it gives you before you rely on it.
- **Let it say "I don't know".** Tell the model in the prompt that it may, and build your app to show and handle that answer, not retry until something else comes back.
- **Do not count on temperature.** Turning it down makes a wrong answer steady, not right.
- **Keep a small set of questions whose answers you know**, run it after every change to the prompt or the model, and count the misses (a [later lesson on test sets](https://spicyshark.co/learn/check-ai-answers-with-a-test-set/) covers how). The guide ends with the same warning: its techniques reduce hallucinations and do not remove them.

## Go deeper

*Optional. This part is for readers who write Python. The code is in the `code/` folder next to this lesson. `python3 show_numbers.py` prints every number below, and `python3 -B -m pytest -q -p no:cacheprovider code/` runs the tests.*

### The toy: count what follows what

The toy has no neural network. It splits text into words and the marks `. , ! ?` (`tokenize`), then counts, for every pair of words and for every single word, which word came next. Every sentence starts after two `<s>` marks, so the first word of a line also follows something.

```python
def train(lines):
    pairs, singles, vocab = {}, {}, set()
    for line in lines:
        words = [START, START] + tokenize(line)
        vocab.update(words[2:])
        for i in range(2, len(words)):
            for table, context in ((pairs, tuple(words[i - 2:i])), (singles, (words[i - 1],))):
                followers = table.setdefault(context, {})
                followers[words[i]] = followers.get(words[i], 0) + 1
    return {"pairs": pairs, "singles": singles, "vocab": sorted(vocab)}
```

To answer, it blends three estimates from vague to specific: a flat guess (every word of the corpus equally likely), what followed the last word, and what followed the last two words. Each blend uses a standard smoothing rule called Witten-Bell: the trust in a context is `seen / (seen + kinds)`, where `seen` is how many words followed it and `kinds` is how many different ones. A context followed by 31 words of 19 kinds earns trust 31/50 = 0.62, and the other 0.38 of the weight goes to the vaguer estimate below it.

```python
def next_word_probs(model, prompt):
    words = [START, START] + tokenize(prompt)
    vocab = model["vocab"]
    if not vocab:
        return {}
    probs = {w: 1 / len(vocab) for w in vocab}
    for table, context in ((model["singles"], tuple(words[-1:])), (model["pairs"], tuple(words[-2:]))):
        followers = table.get(context)
        if not followers:
            continue
        seen = sum(followers.values())
        trust = seen / (seen + len(followers))      # Witten-Bell: seen / (seen + kinds of follower)
        probs = {w: trust * followers.get(w, 0) / seen + (1 - trust) * probs[w] for w in vocab}
    return probs
```

(Docstrings are left out here.) The probabilities add up to 1, and a test checks that.

### One probability by hand

Two numbers from the opening, worked step by step. The first table is the plain corpus and the word Paris. The second adds one copy of "The capital of Peru is Lima." and follows the word Lima.

| step | what | value |
|---|---|---|
| 1 | different words in the corpus, so the flat guess is 1/77 | 0.0130 |
| 2 | words that followed "is" | 31 |
| 3 | different words that followed "is" | 19 |
| 4 | trust in "is": seen / (seen + kinds) = 31/50 | 0.62 |
| 5 | times Paris followed "is" | 5 |
| 6 | one-word estimate: 0.62 x 5/31 + 0.38 x 1/77 | 0.1049 |
| 7 | words that followed "Peru is": never seen, so skip | none |
| 8 | final chance | 0.1049 |

| step | what | value |
|---|---|---|
| 1 | different words in the corpus, so the flat guess is 1/78 | 0.0128 |
| 2 | words that followed "is" | 32 |
| 3 | different words that followed "is" | 20 |
| 4 | trust in "is": 32/52 | 0.6154 |
| 5 | times Lima followed "is" | 1 |
| 6 | one-word estimate: 0.6154 x 1/32 + 0.3846 x 1/78 | 0.0242 |
| 7 | words that followed "Peru is": 1 word, 1 kind, so trust 1/2 | 0.5 |
| 8 | final chance: 0.5 x 1 + 0.5 x 0.0242 | 0.5121 |

Because "Peru is" was never read, the toy skips it and the answer rests on the word "is" alone, which is why Peru is treated like any other country. After one added sentence "Peru is" has a single follower, Lima, and half of the weight goes to "Lima, every time". That is why one sentence moves Lima from 0.0% to 51.2%.

### Adding true sentences: the smallest k

The code adds `k` copies of the true sentence and asks how few are enough. `answer` returns the top word, or "I don't know" when even the top word is under a bar. `smallest_k` tries `k = 0, 1, 2, ...` until the answer is the true word.

```python
def answer(probs, threshold=0.0):
    if not probs:
        return IDK
    word, p = ranked(probs)[0]
    return word if p >= threshold else IDK


def smallest_k(lines, prompt, true_sentence, truth, threshold=0.0, temperature=1.0, limit=50):
    for k in range(limit + 1):
        probs = with_temperature(next_word_probs(model_with_extra(lines, true_sentence, k), prompt), temperature)
        if answer(probs, threshold) == truth:
            return k
    return None
```

| k | chance of Lima | chance of Paris |
|---|---|---|
| 0 | 0.0% | 10.5% |
| 1 | 51.2% | 5.1% |
| 2 | 68.1% | 3.3% |
| 3 | 76.5% | 2.4% |
| 4 | 81.5% | 1.9% |
| 5 | 84.9% | 1.6% |
| 6 | 87.3% | 1.3% |
| 7 | 89.1% | 1.1% |
| 8 | 90.4% | 1.0% |

With no bar, `smallest_k` is 1 for Peru, and also for Laos and Malta. If the toy may answer only when its top word has at least 80%, the answer is 4 (81.5%, where 3 copies give 76.5%). At 90% it is 8.

![Lines for the chance of Lima and of Paris against the number of added true sentences, 0 to 8. Lima rises from 0.0% to 51.2% at 1, 81.5% at 4 and 90.4% at 8. Paris falls from 10.5% to 1.0%.](https://spicyshark.co/learn/why-ai-makes-things-up/figs/adding-true-sentences.svg)

One sentence is enough here only because the toy repeats what it read word for word, and counts a fact written in other words as unseen. A real model keeps a blurred copy of its text, and facts it saw rarely are the ones it gets wrong most (Kandpal and colleagues link accuracy to how often a fact appears; Kalai and colleagues put a floor under errors for facts seen exactly once). Read `k` as a way to see the direction of the effect, not as a number to carry over to real models.

### An "I don't know" bar

With the bar in `answer`, the toy declines when it is not sure. These are the results for the Peru question with no sentence added.

| bar | the toy says |
|---|---|
| 5% | Paris |
| 10% | Paris |
| 11% | I don't know |
| 30% | I don't know |
| 50% | I don't know |

At a 50% bar the toy answers Lima after one added copy, and it still answers Paris to the France question, where it has 85.1%. The bar works because the toy is unsure while it guesses (10.5%) and sure when it has read the fact five times. The cost is that a fact it has read only a little stays under a high bar: at 90% it needs 8 copies.

Kalai and colleagues propose the same idea for tests. The question states a confidence target `t`, a right answer scores 1, "I don't know" scores 0, and a wrong answer costs `t / (1 - t)` points: for `t` = 0.5 the penalty is 1 and for 0.9 it is 9 (two of the paper's own examples). Answering has a positive average score only when the chance of being right is above `t`: `p - (1 - p) x t/(1 - t)` is above 0 exactly when `p` is above `t`. A real model's confidence need not match how often it is right, so any bar like this has to be tested against known answers.

### Temperature in the toy

`with_temperature` raises every chance to the power `1 / temperature` and scales the results back to a total of 1.

```python
def with_temperature(probs, temperature):
    powered = {w: p ** (1 / temperature) for w, p in probs.items()}
    total = sum(powered.values())
    return {w: p / total for w, p in powered.items()}
```

| temperature | Paris | at a 50% bar |
|---|---|---|
| 0.1 | 98.3% | Paris |
| 0.2 | 80.1% | Paris |
| 0.3 | 57.4% | Paris |
| 0.5 | 30.3% | I don't know |
| 1.0 | 10.5% | I don't know |
| 1.5 | 5.9% | I don't know |
| 2.0 | 4.2% | I don't know |

Lima stays at 0.0% because zero to any power is zero: no setting can lift a word that has no chance. The last column shows a trap. The bar is checked after the temperature is applied, so at 0.3 Paris reaches 57.4% and passes the 50% bar, and an honest "I don't know" turns into a confident wrong answer. Checking the bar on the counted chances instead would avoid this in the toy; the point is that a firmer-sounding answer is not evidence of a better one.

### Proof it works

```bash
python3 -B -m pytest -q -p no:cacheprovider code/
```

The tests check these claims. The chances add up to 1 for seven prompts (including an empty one and one with unseen words) at three amounts of added text, and after a temperature change. On the thin corpus the top word is Paris at 10.49% for Peru, Laos and Malta alike, the true word has 0%, and neither it nor the pair "Peru is" appears anywhere in the corpus. Hand calculations match the two tables above for both Paris (0.62 x 5/31 + 0.38/77) and Lima. The smallest `k` is 1, and 4 and 8 at the 80% and 90% bars. The bar and the search limit are each tested on both sides of the boundary, so the search returns `None` when the limit is one too low. The bar gives "I don't know" at 50% with no copy, still answers Paris at 10%, answers Lima after one copy and Paris for France. A lower temperature raises Paris and never the true word, and temperature 1 changes nothing. Ties go to the word that sorts first, whatever the order of the corpus, and an empty corpus gives "I don't know". The tokenizer test feeds in `Peru's capital, Lima! Café 2026?` and checks that the accented letter is skipped (`Caf`), because only the plain letters, digits, apostrophes and four marks are read.

`widget.html` runs the same arithmetic in JavaScript, so a second test, `test_widget.py`, runs the widget's own code in Node and compares it with the Python code: the tokenizer, the vocabulary, every chance to 12 decimal places, the ranking, the answer and the smallest `k`, on five corpora, ten prompts and several values of `k`, temperature and bar (more than a thousand comparisons). One of the corpora is a single word with a bar of 100%, which checks the case where a chance sits exactly on the bar. `embed_widget.py --check` fails if the corpus or the questions in the widget differ from `corpus.txt` and `prompts.py`.

### Where this is used in practice

The public record is mostly papers and documentation. Kalai and colleagues link the pretraining side (facts with no pattern to learn, a share of which appear only once) to the grading side (tests that give "I don't know" nothing). Their example is a what-if, not a measurement: if 20% of birthday facts appear exactly once, a calibrated model should get at least 20% of birthday facts wrong. They also note that hallucination is unavoidable only for a model straight out of pretraining: a system that answers a fixed set of questions from a database, and otherwise says "I don't know", would not make things up. Kandpal and colleagues count the documents in a training set that mention the entities of a question and relate that count to the model's accuracy. Anthropic's guide lists techniques for builders: permission to say "I don't know", quotes before answers, citations with retraction, restricting the model to the documents provided, and running the same prompt several times to compare, since inconsistent answers can point to a made-up one.

## Skills this lesson builds

- Explain why a fluent, confident answer is not evidence that it is true, and name the facts most at risk: rare names, exact numbers, dates, citations and recent events.
- Write a prompt that supplies the source text, asks for quotes and allows "I don't know", and handle that answer in an app.
- Check a quote or a citation by code: find the quote in the source, and open or look up the cited work.
- Compute a smoothed next-word estimate by hand and in code, and read what a confidence bar does to the answers.
- Build a small known-answer test and report right, wrong and "I don't know" counts.

**Portfolio project:** build a command-line tool that asks any chat model 30 questions with known answers, half about common facts and half about rare ones, in two set-ups: with no help, and with the source text in the prompt and "I don't know" allowed. It prints a table of right, wrong and "I don't know" counts for each set-up. Publish the questions, the scoring rule and the table.

## Check yourself

1. A chatbot gives you a study title, authors and a year, and you cannot find the study anywhere. How can that happen, and what do you do before using it?

   <details><summary>Answer</summary>The model writes the likely next words. Where the training text says little, such as a rare title or an exact citation, a smooth, well-formatted continuation is still likely and can be wrong, and nothing in the basic design checks it against a source. Open the cited work, or give the model the real source text and ask it to quote from it.</details>

2. You turn the temperature to its lowest value to stop invented answers. What changes, and what does not?

   <details><summary>Answer</summary>The answers become more repeatable, because the model takes its top choice almost every time. They do not become truer: the top choice is taken whether or not it is true, and a fact the model gave no chance still gets none. Giving it the source text, allowing "I don't know" and checking the answers against known ones help instead.</details>

3. A word is followed by 9 words in all in the text, 3 of them different, and one of the three followed it 3 times. The text has 50 different words, and only this one-word context applies. With the toy's rule, what is the chance of the word that followed it 3 times?

   <details><summary>Answer</summary>Trust is seen / (seen + kinds) = 9 / (9 + 3) = 0.75. The chance is 0.75 x 3/9 + 0.25 x 1/50 = 0.25 + 0.005 = 0.255.</details>

4. With a bar of 80%, after 3 added copies the toy has 76.5% on Lima. What does it say, and how many copies does it need?

   <details><summary>Answer</summary>It says "I don't know", because its top word, Lima, is below the bar. It needs 4 copies, where Lima has 81.5%.</details>

5. At temperature 1 and a 50% bar the toy says "I don't know" to the Peru question, but at 0.3 it says Paris. Why, and why does Lima not rise?

   <details><summary>Answer</summary>A lower temperature sharpens the chances, so Paris goes from 10.5% to 57.4% and passes the bar, which is checked after the temperature is applied. Lima does not rise because its chance is 0, and 0 to any power is still 0.</details>

## Further reading

**Gentle**

- [Reduce hallucinations](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations) (Anthropic documentation; read 2026-10-19)
- [What are AI hallucinations?](https://www.ibm.com/think/topics/ai-hallucinations) (IBM explainer; read 2026-10-19)

**Deep**

- [Why Language Models Hallucinate](https://arxiv.org/abs/2509.04664) (Kalai, Nachum, Vempala and Zhang, 2025; read 2026-10-19)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://arxiv.org/abs/2211.08411) (Kandpal, Deng, Roberts, Wallace and Raffel, 2022; abstract read 2026-10-19)