A toy. It only counts which word follows which in the sentences it has read. It has no idea what a capital is, and it is nothing like the size or the insides of a real chatbot.
The toy reads the last two words of the start of the sentence, looks at what followed them in the text, then at what followed the last word alone, and blends the two with a flat guess. This is the same arithmetic as code/toy_lm.py (the Witten-Bell rule), so the numbers match the lesson.
It repeats sentences it has seen word for word, and a fact written in other words counts as unseen. A real model keeps a compressed, blurred version of its training text, and can often answer a question it never saw in those exact words.
Capital letters count: “Peru” and “peru” are different words. Only letters, digits, apostrophes and the marks . , ! ? are read. Temperature raises each probability to the power 1/T and scales the result back to total 1. The “I don’t know” check uses the probabilities after the temperature.