Byte-level BPE tokenizer

Training the tokenizer on the lesson's corpus…

1. Text

Text to split (up to 4,000 characters)

The samples are not in the training corpus.

2. Merges learned

Merges used (0 to 512)

Drag the slider to the right, or press +10. Each merge joins the pair of neighbours that is most common in the corpus.

3. Tokens

Show ids and byte counts

· space, ↵ newline, → tab. Dashed: raw bytes in hex.

Same tokenizer, four samples

Table of counts

Sample	Tokens	Bytes per token	Tokens per word

Cost of one request

Prices in dollars per million tokens (examples: use your provider's)

Input

Cached

Output

Tokens (new = not cached)

New

Cached

Output

Requests

Part	Tokens	Dollars	Share

Work (a model, not a measurement)

Assumptions and limits

The tokenizer is trained in this page on a corpus of about 15 KB written for the lesson, with the same splitting pattern and the same tie-break as code/bpe.py, so its counts match the Python code's. It is a toy: a production tokenizer is trained on far more text, and has a vocabulary of tens or hundreds of thousands of entries, not 768. Its sizes show direction, not what any provider's model will count. Count real prompts with the provider's own tokenizer.

Words are runs of text between spaces, so a Korean word is a space-separated group of syllables. The Korean sample was written for the lesson and has not been reviewed by a native speaker. Only the first 1,000 tokens are drawn.

The prices are examples you replace. The bill follows the lesson's request_cost: three rates, no charge for writing the cache, no minimum cacheable length, no discount for batches. The work lines follow sequential_passes and decode_positions_read: positions, not bytes or seconds.