Tokens and tokenization

After this lesson you can: Explain why an LLM cannot reliably count the letters in a word.

Watch

• Why AI Tokens are so Expensive - Computerphile - Computerphile, 25 min. Directly targets tokens, with practical cost framing familiar to engineers deploying API-backed systems.
• Glitch Tokens - Computerphile - Computerphile, 19 min. Tokenization edge cases make the “models read tokens, not text” misconception concrete.
• [1hr Talk] Intro to Large Language Models - Andrej Karpathy, 60 min. A concise canonical LLM overview that can place tokenization inside the whole prediction pipeline.

Notes

An LLM does not receive a string the way your program does. Before the model sees text, a tokenizer converts it into a sequence of token IDs. A token might be a whole common word, part of a word, punctuation, whitespace, or a byte-like fragment. The model then predicts the next token ID, not the next character and not exactly the next English word.

Unlock the full lesson notes, the exercises and the graded checkpoint - ₹5,000 once.