01.07 · Walkthrough
Tokenisation at Serving Time
Diagnose tokenisation issues in production: token count vs character count, non-English compression, and special token handling at the serving boundary.
Tokenisation failures at serving time usually come from treating characters as a proxy for model tokens. The server admits, schedules and truncates by token IDs after normalisation, chat templating and special-token insertion, so multilingual input, duplicated markers and tokenizer mismatches can break requests that looked safe in the UI.
What this lesson answers
- why token count differs from character count
- how chat templates duplicate special tokens
- why Chinese prompts exceed context limits
Notes
Tokenisation at serving time is the boundary transform where raw request text becomes token IDs, with serving cost and context usage determined by , not by character count. Algorithm: normalize/pre-tokenize string apply BPE/SentencePiece/Unigram merges map pieces to integer IDs optionally add special tokens such as BOS/EOS/chat markers; then enforce .
Common questions
- Why does a prompt fit by character count but fail at inference?
- The model never sees characters directly. Serving code converts the rendered request into token IDs, then checks the prompt tokens plus the requested generation budget against the model limit. If the frontend estimated length from characters, it can undercount badly, especially for non-English text or after chat template and special token insertion.
- Why do different servers report different token counts for the same text?
- Token counts depend on the exact tokenizer, its revision, normalisation rules, chat template and whether special tokens are added. A GPT-style tokenizer, a Llama-style tokenizer and a Qwen-style tokenizer can split the same Unicode string differently. Production logs need the tokenizer identity and final rendered prompt, not just the raw user text.
- How do special tokens cause serving bugs?
- Special tokens are often added at the serving boundary: beginning and end markers, instruction tags, assistant markers or chat-role wrappers. If a client sends preformatted chat text and the server applies its own template as well, those markers can be duplicated or placed in the wrong order, changing both token count and model behaviour.
Short definition: what is Tokenisation at Serving Time?