---
title: "What is a token, and why is AI priced per token?"
description: "A token is a chunk of text, often a word or part of one, that AI services bill by, on what you send and what comes back. Counts differ by language and content."
topic: "AI systems"
chapter: "how-ai-answers"
order: 1
minutes: { plain: 8, deep: 14 }
prerequisites: ["Reading short Python functions", "Bytes and UTF-8 (one character can take 1 to 4 bytes)"]
seo_title: "What is a token, and why is AI priced per token?"
date: 2026-10-18
---
# What is a token, and why is AI priced per token?
AI services bill by the token, not by the word or the minute. A token is a short chunk of text, often a word or part of one, and how much text one token holds varies. The Korean sample in this lesson is 92 characters long. One published way of cutting text into tokens turns it into 201 tokens, and another published way turns the same sample into 54. At one price per token, the first count costs 3.7 times as much (201 / 54). This lesson builds a small program that cuts text into tokens, small enough to read in one sitting, and shows why the work a model does follows the token count, which is a plausible reason for the price.
## In one minute
Picture a print shop that sells text by the block. A block is a short run of letters, such as `The`, ` at` or `er`. The shop keeps a catalog of blocks, and every block has a number. To set your text, the shop first spells it in the smallest blocks, then glues neighbors together wherever its glue list says to. When no glue fits, the blocks left are your tokens, and the shop bills by that count.
The same text can need more blocks or fewer, because the glue list comes from the text the shop studied: common English glues into big blocks, and text the shop saw less of stays in small ones. Price plausibly follows tokens because work follows tokens (providers do not state their reason). The model reads all your blocks in one pass, but writes its reply one block at a time, and each block waits for the one before it. Providers publish separate rates for input and output, and a lower rate for input reused from an earlier request. This lesson counts work, not dollars.
## Explore the numbers
[Open the widget](https://spicyshark.co/learn/what-is-a-token/widget.html) to cut your own text into tokens and to price a request. It trains the lesson's tokenizer in your browser, on a 15 KB corpus written for the lesson.
- **Cut your text.** Pick English, Code, Numbers or Korean, or paste your own. Move the merges slider from 0 to 512; it starts at 0, where every byte is a token. At 512 the English sample is 101 tokens for 224 bytes.
- **Price a request.** Type three example prices (dollars per million tokens) and three counts for one request: new input, input read from a cache (priced lower), and output. With the starting values (2, 0.2 and 10 dollars; 500, 1,500 and 400 tokens) one request costs 0.0053 dollars, and 1,000 requests cost 5.30 dollars against 8.00 with no cache.
The prices are examples to replace, not a price list. Go deeper says how the page and `code/bpe.py` differ.
## How it works
### The picture, step by step
The print shop, one step at a time. The next part shows the line of code behind each step, and the part after that runs the steps on a real sentence.
1. **Learn the glue list, once.** Before it opens, the shop studies a big pile of text. It finds the two blocks that sit side by side most often, glues them into one new block, and writes "glue these two" on a list. Then it looks again and repeats. The list stays in the order learned, so the most common glues come first, and it never changes after that. (Engineers call this byte pair encoding, or BPE, and the shop is the tokenizer.)
2. **Cut at word edges, then spell each piece small.** Your text arrives. The shop cuts it into pieces at word edges, so no glue ever joins two words; a piece keeps the space in front of it. Then it spells each piece in the smallest blocks there are, single bytes. A plain English letter is one byte, and the Korean syllable `한` is three.
3. **Find the earliest glue that fits.** Inside a piece, the shop checks every pair of neighboring blocks against the glue list and takes the one learned first.
4. **Glue it, then look again.** The pair becomes one block with a new number, and the shop checks again. When nothing on the list fits, it stops. The blocks left are the tokens, and the model receives their numbers (ids), not the letters.
5. **Bill by the block, at three rates.** One rate for new blocks you send, a lower one for blocks the shop already set for an earlier job, and a separate one for blocks it writes back (in this lesson's example prices, the highest). Only the start of a job can be reused: it must match the earlier job from the very first block.
6. **Read all at once, write one block at a time.** The shop takes in all your blocks in a single pass. Each block of the reply depends on the one before it, so the steps in a row equal the length of the reply.
### Each step in code
One real line of the lesson's code does each step. The files are in `code/`, and the line numbers are those of this lesson's files.
| Step | The code line | What it does |
|---|---|---|
| 1. Learn the glue list | `bpe.py:58`
`best = min(pairs, key=lambda pair: (-pairs[pair], pair))` | `pairs` holds how often each pair of neighbors occurs in the pile. This line picks the most frequent one, and on a tie the smaller pair of ids. A loop around it repeats up to 512 times, and each pick is one entry on the glue list. |
| 2. Cut, then spell small | `bpe.py:107`
`word = list(piece.encode("utf-8"))` | `piece` is one cut piece (`pieces()` at `bpe.py:37-39` does the cutting). This line turns it into a list of its bytes, each a number from 0 to 255. |
| 3. Find the earliest glue | `bpe.py:109`
`best = min(zip(word, word[1:]), key=lambda pair: rank.get(pair, len(rank)))` | `zip(word, word[1:])` lists every pair of neighbors. `rank` gives each pair its place on the glue list, and a pair not on the list gets a place past the end. `min` takes the pair learned first. |
| 4. Glue it, look again | `bpe.py:112`
`word = list(_apply(word, best, 256 + rank[best]))` | Replaces that pair with one block whose id is 256 plus its place on the list. The `while` on line 108 runs again, and lines 110 and 111 stop it when no pair is on the list. What is left in `word` is the tokens. |
| 5. Bill by the block | `cost.py:26`
`total = counts[0] * prices[0] + counts[1] * prices[1] + counts[2] * prices[2]` | The counts are new input, cached input and output, and the prices are the three matching rates. Line 27 then divides by one million, because prices are per million tokens. |
| 6. Count the steps in a row | `cost.py:39`
`return max(1, output_tokens)` | The number of passes that cannot overlap is the length of the reply, and at least one. The whole prompt goes through in the first pass. |
### A real run
`trace.py` runs the steps on a short sentence. It calls the same `encode`, `request_cost` and `sequential_passes` as the table, and only prints what they do, one glue at a time. Run it from the repository root:
```bash
cd lessons/what-is-a-token/code && python3 trace.py
```
```text
text 'The ferry leaves at nine': 24 bytes, 5 pieces, 512 merges allowed
piece 'The': bytes [84, 104, 101]
merge 1: 104 + 101 -> 257 b'he', now [84, 257]
merge 87: 84 + 257 -> 343 b'The', now [343]
piece ' ferry': bytes [32, 102, 101, 114, 114, 121]
merge 7: 101 + 114 -> 263 b'er', now [32, 102, 263, 114, 121]
merge 15: 32 + 102 -> 271 b' f', now [271, 263, 114, 121]
merge 331: 114 + 121 -> 587 b'ry', now [271, 263, 587]
piece ' leaves': bytes [32, 108, 101, 97, 118, 101, 115]
merge 14: 101 + 115 -> 270 b'es', now [32, 108, 101, 97, 118, 270]
merge 33: 32 + 108 -> 289 b' l', now [289, 101, 97, 118, 270]
piece ' at': bytes [32, 97, 116]
merge 4: 32 + 97 -> 260 b' a', now [260, 116]
merge 234: 260 + 116 -> 490 b' at', now [490]
piece ' nine': bytes [32, 110, 105, 110, 101]
merge 12: 105 + 110 -> 268 b'in', now [32, 110, 268, 101]
merge 34: 32 + 110 -> 290 b' n', now [290, 268, 101]
merge 150: 268 + 101 -> 406 b'ine', now [290, 406]
tokens: 12, from 24 bytes
blocks: 'The' ' f' 'er' 'ry' ' l' 'e' 'a' 'v' 'es' ' at' ' n' 'ine'
price per million tokens (examples): new input 2, cached input 0.2, output 10 dollars
500 new input at 2 = 0.001 dollars
1,500 cached input at 0.2 = 0.0003 dollars
400 output at 10 = 0.004 dollars
total 0.0053 dollars; with nothing cached 0.008 dollars
sequential passes (the same 2,400 tokens, split two ways)
2,000 in + 400 out: 400 passes in a row, 877,800 positions read
400 in + 2,000 out: 2,000 passes in a row, 2,798,600 positions read
```
Read it against the six steps.
- **Steps 1 and 2.** The sentence is 24 bytes, and the shop cuts it into 5 pieces. Each piece first shows as raw bytes: ` ferry` is 6, because the space in front is byte 32. The glue list has 512 entries. Only the 12 that fit this sentence are printed, each with its place on the list (`merge 15`) and the id it creates, which is 256 plus that place.
- **Steps 3 and 4.** Take ` at`, bytes `[32, 97, 116]`. The earliest glue that fits is merge 4, `32 + 97`, so ` a` comes first. Then `260 + 116` fits, as merge 234, and one block is left: ` at`. A glue can use a block an earlier glue made, which is why `The` takes two glues: merge 1 makes `he`, and merge 87 joins `T` to it.
- **The count.** Each glue turns two blocks into one. 24 bytes minus 12 glues leaves 12 tokens, which is the `tokens:` line. The `blocks:` line shows what they are. ` leaves` takes only two glues, `es` and ` l`, so it stays in five blocks.
- **Step 5.** The three lines are 500 x 2 = 1,000 millionths of a dollar, 1,500 x 0.2 = 300 and 400 x 10 = 4,000, which add up to 5,300 millionths: 0.0053 dollars. With nothing cached, all 2,000 input blocks pay the full rate and the request costs 0.008 dollars.
- **Step 6.** The same 2,400 blocks split two ways. Reading 2,000 and writing 400 takes 400 passes in a row. Reading 400 and writing 2,000 takes 2,000. The positions-read counts add up how many earlier blocks the writing passes look back at: each of the `p` passes after the first (`p` is the reply length minus one) reads the whole prompt plus the blocks written so far, and together they read `p x input + p(p + 1) / 2` positions.
The prices are examples, not a price list.
## Your turn
The glue list came from a pile of text with a lot of English and a little Korean. See what that does to Korean.
1. In `code/trace.py`, change line 10 to `TEXT = "한국어"` (the Korean word for "Korean").
2. Run `python3 trace.py` again in the `code` folder.
Look at the `tokens:` line and the `blocks:` line. How many blocks do the three syllables become, out of how many bytes? Are all three syllables glued the same way?
What you should see
The top of the output:
```text
text '한국어': 9 bytes, 1 piece, 512 merges allowed
piece '한국어': bytes [237, 149, 156, 234, 181, 173, 236, 150, 180]
merge 63: 237 + 149 -> 319 b'\xed\x95', now [319, 156, 234, 181, 173, 236, 150, 180]
merge 196: 150 + 180 -> 452 b'\x96\xb4', now [319, 156, 234, 181, 173, 236, 452]
merge 231: 236 + 452 -> 487 b'\xec\x96\xb4', now [319, 156, 234, 181, 173, 487]
merge 432: 319 + 156 -> 688 b'\xed\x95\x9c', now [688, 234, 181, 173, 487]
tokens: 5, from 9 bytes
blocks: '한' b'\xea' b'\xb5' b'\xad' '어'
```
The 9 bytes become 5 tokens. `한` and `어` are one block each, because the glue list joins all three of their bytes. In the pile, `한` appears 10 times and `어` 10 times. `국` appears once, so none of its bytes is glued and it takes three blocks. This is the opening of the lesson in small: the same kind of text takes more blocks when the glue list has seen less of it.
## What this means when you build
- **Estimate cost before you ship.** Take real prompts, count their tokens, add the output length you expect, and multiply by your provider's current rates. Word counts mislead: under the toy, tokens per word were 2.24 for the English sample and 5.35 for the Korean one.
- **Count with the model's own tokenizer, on your own data.** The same text was 201 tokens under one encoding and 54 under another, and code, numbers and non-English text can differ from English prose. Google's token guide describes a `count_tokens` call to make before sending a request.
- **Read the usage fields.** Anthropic's Messages API reports `input_tokens`, `cache_read_input_tokens` and `cache_creation_input_tokens`. OpenAI's Chat Completions API reports cached tokens in `usage.prompt_tokens_details.cached_tokens`, and its Responses API in `usage.input_tokens_details.cached_tokens`. Anthropic's `input_tokens` leaves out cached tokens; OpenAI's includes them, so check each provider's definition before you add or subtract. A cache that is not working shows up as zero.
- **Keep the stable part first.** Put instructions, tool definitions and examples at the start, and anything that changes per request, such as the time or the question, at the end. A time stamp on the first line can make every request a cache miss: in the lesson's toy, two requests with different questions shared 15 of 113 prompt tokens (13%) with the time stamp first and 93 (82%) with it last.
- **Watch output length.** Output tokens each need their own pass and have their own rate. Ask for the format you need and set a maximum.
## Go deeper
*Optional. The reasons, sources and measurements behind the picture, then the code in the `code/` folder. `python3 show_numbers.py` prints every toy number in this section, and `python3 compare_tiktoken.py` prints the published-tokenizer numbers.*
### Training: count pairs, merge the best
Byte pair encoding is a compression method from Gage (1994), adapted to translation by Sennrich and colleagues (2015). Many current tokenizers, including the OpenAI ones measured here, start from bytes. In UTF-8 the letter `a` is 1 byte, `é` is 2, the Korean syllable `한` is 3 and the emoji `😀` is 4 (`show_numbers.py` prints these). Any text in any script is a list of numbers from 0 to 255, so no input is "unknown": Wang, Cho and Gu (2019) studied byte-level BPE, which they describe as having "no out-of-vocabulary tokens". Bytes alone make long lists: the English sample is 224 bytes, so with no merges it is 224 tokens. Merging shortens the list with one step, repeated: count every pair of neighboring ids in a training text, give the most frequent pair a new id, replace it everywhere, and repeat.
`train_counted` in `code/bpe.py` is the whole of training. `words` counts how often each piece, as a tuple of ids, occurs:
```python
words = Counter(tuple(p.encode("utf-8")) for p in pieces(text))
learned = []
for _ in range(num_merges):
pairs = Counter()
for word, freq in words.items():
for pair in zip(word, word[1:]):
pairs[pair] += freq
if not pairs:
break
best = min(pairs, key=lambda pair: (-pairs[pair], pair))
if pairs[best] < min_count:
break
learned.append((best, pairs[best]))
words = _merge_words(words, best, 255 + len(learned))
```
Two rules make it deterministic, and the widget copies both: a tie in count goes to the smaller `(left_id, right_id)`, and training stops when the best pair is seen fewer than `MIN_COUNT = 2` times.
By hand on `aaabdaaabac`: `aa` is seen 4 times (twice in each `aaa`), so merge 0 is `aa`. The string is now `256 97 98 100 256 97 98 97 99`, where `ab` is seen twice: merge 1. Then `256 257` is seen twice: merge 2. After that every pair is seen once. On the lesson's corpus (14,553 characters, 15,793 bytes, 3,735 pieces) the first merges are ` t` seen 371 times, `he` 267, two spaces 248, ` the` 208 and ` a` 198. Merge 99 is `ken` (20 times) and merge 511 is ` any` (4). The longest entry that is not only whitespace is ` tokenizer`.
The figure follows the string `aaabdaaabac` through the same three merges.
Eleven bytes become five tokens: `[258, 100, 258, 97, 99]`. In the lesson's tokenizer, merge number `k` makes id `256 + k`, so 512 merges give 768 ids. Published tokenizers are far larger: the tiktoken encodings `gpt2`, `cl100k_base` and `o200k_base` have 50,257, 100,277 and 200,019 ids.
### Encoding and decoding
Encoding is steps 2 to 4 of the picture. Lines 105 to 114 of `code/bpe.py` are the whole loop: `rank` (line 102) maps each pair to its place on the glue list, and `cache` remembers pieces already worked out. Inside each piece, the loop applies the earliest-learned merge that is present, again and again. The result equals replaying all the merges in training order on each piece, and a test checks that.
Before the loop, a splitting pattern cuts the text into pieces (a word with its leading space, a run of digits, a run of punctuation) so that a merge never joins across a word boundary. Each published tokenizer has its own splitting pattern, so the toy's counts are its own.
Decoding joins each id's bytes and calls `decode("utf-8", errors="replace")`, so ids that cut a character in half show as the replacement character.
### Why pieces, and not letters or words
A list of single bytes is too long. The work a model does grows with the length of its input, and attention (the step where each token looks back at earlier ones) compares every token with every earlier one; [Why does a chatbot forget?](https://spicyshark.co/learn/why-chatbots-forget/) counts those pairs. On the English sample the byte list is 2.22 times as long as the list after 512 merges: 224 tokens against 101.
A table with one entry per word form would be huge and would still meet words it has never seen: names, typos, identifiers, long numbers. Starting from bytes means nothing is unseen: frequent strings get one id and rare strings are spelled out.
How many merges is a trade-off. Token counts for the four samples (none is in the corpus) and for the corpus itself, as merges are added:
| Text | 0 merges | 64 | 128 | 256 | 512 |
|---|---|---|---|---|---|
| English sample | 224 | 144 | 131 | 112 | 101 |
| Code sample | 182 | 125 | 101 | 76 | 70 |
| Numbers sample | 157 | 138 | 128 | 117 | 102 |
| Korean sample | 228 | 202 | 172 | 147 | 123 |
| The corpus | 15,793 | 10,956 | 9,593 | 8,171 | 6,760 |
Later merges save less. Training takes the most frequent pair first, so the pair counts on the training text never rise, and a merge saves its count except when its pair overlaps itself: three spaces hold two pairs of spaces, but one merge saves one token (this happens in 5 of the 512 merges). On average, between 0 and 64 merges each saves 75.58 tokens on the corpus the tokenizer learned from, and between 256 and 512 each saves 5.51. On the four samples together, which it has not seen, the averages are 2.84 and 0.22. Meanwhile each merge adds a row to the embedding table (the numbers the model looks up for each id) and one more score at the model's output (Vaswani and colleagues, 2017, section 3.4); Go deeper has the sizes.
### Why the same text has different counts
Three published encodings and the toy count the same four samples like this:
| Tokenizer | English | Code | Numbers | Korean |
|---|---|---|---|---|
| toy (512 merges) | 101 | 70 | 102 | 123 |
| `gpt2` (50,257 ids) | 51 | 79 | 64 | 201 |
| `cl100k_base` (100,277 ids) | 51 | 54 | 79 | 89 |
| `o200k_base` (200,019 ids) | 51 | 54 | 79 | 54 |
- **English prose** takes 51 tokens in all three published encodings: 4.39 bytes per token.
- **Numbers** took more tokens under `cl100k_base` and `o200k_base` than under `gpt2` (79 against 64), although their vocabularies are larger: those two encodings cut every run of digits into groups of at most three before any merge. Under `cl100k_base` the date `20260418` in the Numbers sample is cut into `'202'`, `'604'`, `'18'`.
- **Korean** takes 201 tokens under `gpt2` for 228 bytes, barely better than one token per byte (1.13 bytes per token). Under `o200k_base` it takes 54 (4.22).
A merge exists only if its pair was frequent in the tokenizer's training text. The toy shows the effect: Hangul is 11.8% of the bytes in its corpus, and a Korean syllable is 3 bytes that must be merged twice to become one token, so the toy gets 1.85 bytes per token on Korean and 2.22 on English. What a published tokenizer was trained on, this lesson cannot check.
Researchers have measured the gap. Petrov and colleagues (2023) report that the same text translated into different languages can have "differences up to 15 times in some cases" in tokenization length. Ahia and colleagues (2023) study what this means for API pricing by tokens. The tiktoken README says that "on average, in practice, each token corresponds to about 4 bytes"; our samples give 4.39 for English, 3.37 for code at best, and 1.13 to 4.22 for Korean. A model reads ids and not letters, which is one reason models often miscount the letters in a word.
### Why work follows tokens
A model writes one token at a time, and to choose each it reads the whole text before it. Serving a request has two phases. In **prefill**, all the input tokens are known in advance, so the model processes them together in one pass, which also yields the first output token. In **decode**, each later output token depends on the one just before it, so each needs a pass of its own, and passes cannot overlap. The [paged KV cache lesson](https://spicyshark.co/learn/paged-kv-cache/) covers the stored notes that decode reads.
`sequential_passes` in `code/cost.py` counts this: the output length sets the number of passes, with a minimum of one. A request with 2,000 input and 400 output tokens needs 400 passes in a row. One with 400 input and 2,000 output, the same 2,400 tokens in all, needs 2,000. A second function, `decode_positions_read`, adds up the earlier positions the decode passes look back at: 877,800 and 2,798,600 for those two requests. It counts only earlier positions read, not the model weights each pass also reads. These are counts in a simple model, not seconds or dollars.
Patel and colleagues (2023) describe the two phases as "a compute-intensive prompt computation, and a memory-intensive token generation". That fits pricing input and output separately. The Anthropic and Google pricing pages, read on 2026-10-18, list an input rate and an output rate per million tokens, and in the rows checked for this lesson the output rate is the higher one. Neither page explains the ratio, so the work counts are a reason that fits, not the providers' stated reason.
### Why cached input costs less
If two requests start with the same text, the server can keep the stored notes for that shared start (the KV cache, covered in [Why does a chatbot forget?](https://spicyshark.co/learn/why-chatbots-forget/)) and read them back instead of computing them again. OpenAI's prompt caching guide (read 2026-10-18) says the cache stores key-value tensors and that the model "still needs to process any new input". Providers bill the reused part at a lower rate: Anthropic's pricing page says the API reads from cache at a fraction of the standard input price. Some providers charge extra to write the cache, and both Anthropic and OpenAI set a minimum prefix length below which nothing is cached.
Only a shared *prefix* can be reused. Anthropic's caching guide says cache hits require "100% identical prompt segments" and advises putting static content at the beginning of the prompt. The toy shows the effect. Take a bakery assistant's instructions (31 words), a time stamp and a question. Two requests with different questions, seven seconds apart, share 93 of the first prompt's 113 tokens (82%) when the instructions come first, and 15 (13%) with the time stamp first, because the first difference ends the shared prefix.
At the example rates of 2, 0.2 and 10 dollars per million tokens, 500 new input, 1,500 cached input and 400 output tokens cost 0.001 + 0.0003 + 0.004 = 0.0053 dollars. Output is 16.7% of the tokens (400 of 2,400) and 75.5% of the cost. With nothing cached the same request costs 0.008 dollars, 1.51 times as much.
### How the widget matches the code
When it loads, the widget trains this lesson's tokenizer (512 merges on the 15 KB corpus). It uses the same corpus and the same rules as `code/bpe.py` (the splitting pattern and the tie-break rule above), and a test compares the two on 505 text-and-merge-count pairs, including CRLF line endings, combining marks and a 4,000-character text. The page differs from `bpe.py` in two ways:
- The text box turns CRLF line endings into LF, so pasted text with CRLF can count differently from `bpe.py` on the same text with its CRLF kept.
- The box stops at `maxlength=4000`, which counts UTF-16 code units (an emoji counts as 2), and it cuts a longer paste without a message.
The count covers all of the text that fits, but only the first 1,000 tokens are drawn.
### Compare with published tokenizers
`pip install tiktoken`, then `python3 compare_tiktoken.py`. The first run of each encoding downloads its vocabulary file, so it needs network access once; the script says "skipped" and carries on if it cannot. It prints the token counts in the table above, the most digits any one token holds in the Numbers sample (4 under `gpt2`, 3 under the other two, whose splitting patterns cap a digit run at three), and this table:
| Bytes per token | English | Code | Numbers | Korean |
|---|---|---|---|---|
| toy (512 merges) | 2.22 | 2.60 | 1.54 | 1.85 |
| `gpt2` | 4.39 | 2.30 | 2.45 | 1.13 |
| `cl100k_base` | 4.39 | 3.37 | 1.99 | 2.56 |
| `o200k_base` | 4.39 | 3.37 | 1.99 | 4.22 |
Tokens per word is a poor unit for Korean, so prefer bytes per token when you compare languages. The Korean sample was written for this lesson and has not been reviewed by a native speaker.
### What a bigger vocabulary adds to the embedding table
Each id has one row of numbers in the embedding table. At an example width of 4,096 numbers per row, not any model's, the toy's 768 rows hold 3,145,728 numbers and 200,019 rows hold 819,277,824. `show_numbers.py` prints these and the sizes for `gpt2` and `cl100k_base`.
### The bill in code
`request_cost` uses `Decimal`, so cents are exact, and prices are dollars per million tokens:
```python
total = counts[0] * prices[0] + counts[1] * prices[1] + counts[2] * prices[2]
return total / PER
```
By hand: 500 x 2 + 1,500 x 0.2 + 400 x 10 = 5,300 millionths, which is 0.0053 dollars. It leaves out a charge for writing the cache, a minimum cacheable length, batch discounts and prices that change with prompt length; add those from your provider's documentation. `common_prefix_len` counts the leading tokens two prompts share, the part a cache can reuse.
### Proof it works
```bash
cd lessons/what-is-a-token/code && python3 -m pytest -q
```
The 34 tests check the hand-computed examples (`aaabdaaabac`, `ababab` and the bills), the tie-break and stop rule, that the splitting pattern covers every character once, and that decoding returns the text. One test runs the widget's JavaScript under Node.js (so it needs Node.js installed) on 505 text-and-merge-count pairs and a few bills, and requires the same ids, counts, cost, passes and positions as the Python. Changing the tie-break or the stop rule in either implementation makes a test fail.
## Skills this lesson builds
- Count tokens with a model's own tokenizer and predict a request's cost from counts and rates.
- Explain byte-level BPE, and implement train, encode and decode with a round-trip test.
- Read a response's usage fields and reconcile them with a bill, including cached input (providers define the fields differently).
- Lay out prompts so a stable prefix comes first, and measure the cache hit rate.
- Compare tokenizers on different languages and content.
**Portfolio project:** build a command-line cost estimator. It reads a folder of real prompts, counts tokens with a tokenizer you choose, takes rates from a config file you edit, and reports the cost per request with and without a stable prefix. Publish the method and a test with hand-computed values.
## Check yourself
1. Train the lesson's BPE on the string `ababab` (one piece; `a` is byte 97 and `b` is 98; `MIN_COUNT` is 2). What are the merges, and how is the string encoded?
Answer
The pair `ab` (97, 98) is seen 3 times, so merge 0 makes id 256 and the string becomes 256 256 256. The pair (256, 256) is seen twice, so merge 1 makes id 257, applied left to right: `[257, 256]`. The pair (257, 256) is seen once, so training stops. `train_counted("ababab")` returns `[((97, 98), 3), ((256, 256), 2)]`.Answer
2,000 x 2 = 4,000, 6,000 x 0.2 = 1,200 and 500 x 10 = 5,000, which add up to 10,200 millionths: 0.0102 dollars. With nothing cached, 8,000 x 2 + 5,000 = 21,000: 0.021 dollars, a little over twice as much.Answer
A merge exists only if its pair was frequent in the tokenizer's training text, and a Korean syllable is 3 bytes that need two merges to become one token. If Korean was rare in that text, few such merges exist, and Korean stays near one token per byte (1.13 here). The ratio 201 / 51 divides two different texts, not a text and its translation. The comparison that holds the text fixed is 201 tokens under `gpt2` against 54 under `o200k_base`.Answer
A cache can reuse only a shared prefix, and the time changes in the first tokens of every request, so almost nothing is shared. Move the time after the instructions, next to the question. In the toy, two prompts share 15 of 113 tokens (13%) with the time first and 93 of 113 (82%) with it last. Read the cached-token field in the response to confirm.Answer
Both need 1,000 passes: the number follows the output length, and the whole prompt goes through in one pass. Positions read are `p x input + p(p + 1) / 2` with `p = 999`: 1,498,500 for A and 2,497,500 for B. The longer prompt adds reading to every decode pass but no passes.