Tokenization and Dataset Packing
Tokenization and dataset packing is the data-preparation stage that converts raw text into model vocabulary IDs, measures how long records become, and arranges those IDs into fixed-length training sequences. It determines what the model can consume, how much padding or truncation occurs, and how efficiently training hardware is used.
Language models do not train on characters or words directly; they train on integer IDs from a specific tokenizer. That matters because the same document can become very different sequence lengths depending on how code, punctuation, whitespace, uncommon words, or non-English text are split. Estimating from characters or words is a common mistake. Engineers need corpus-specific token statistics before choosing context length, chunking rules, filtering rules, or storage and compute budgets.
The pipeline first applies a fixed tokenizer to each record, producing a list of token IDs and usually special markers such as an end-of-document token. You then inspect the length distribution and examples at the extremes. Packing takes those tokenised records, concatenates them with the required boundary markers, and cuts the resulting stream into blocks of the model’s target length, optionally carrying attention masks or loss masks alongside the IDs.
Packing trades simplicity for utilisation. Padding every short example to the full sequence length is easy but wastes computation; packing makes tensors dense and uniform. The risk is semantic leakage across document boundaries, incorrect loss on prompt or padding regions, hidden truncation, duplicated separators, or broken Unicode handling. Whether cross-document concatenation is acceptable depends on the objective, dataset type, and masking scheme, not on packing itself.
Engineers meet this in training-data builders, preprocessing jobs, iterable datasets, and framework collators. The practical discipline is reproducibility: version the tokenizer, record special tokens and maximum length choices, and emit metrics such as total tokens, packed sequence counts, padding, truncation, and source mix. A reliable final check is to decode sampled packed blocks back to text and inspect whether boundaries and masks match the intended training examples.
Common questions
- Is tokenization just splitting text into words?
- No. LLM tokenizers usually split text into subword-like pieces and map each piece to an integer ID. A token may be a word, part of a word, punctuation, whitespace, or a byte-level fragment. That is why word counts are a poor proxy for training length.
- Why pack examples instead of padding them?
- Padding makes each example the same length by adding tokens that normally carry no training signal. If many records are short, the model spends compute on padding. Packing fills fixed-length blocks with real token IDs from multiple examples, while using boundary markers and masks to preserve the intended training behaviour.
- Can packing hurt model quality?
- Yes, if implemented carelessly. The model may learn across unrelated document boundaries, compute loss on tokens that should be ignored, or lose important text through truncation. Good packing is not merely concatenation; it includes boundary tokens, masking rules, validation by decoding samples, and metrics that reveal dropped or padded data.
- Do I need to remeasure token lengths when changing tokenizer?
- Yes. Token counts are tokenizer-specific. Changing vocabulary, special tokens, normalisation, or byte handling can alter sequence lengths and therefore affect context fit, storage, batching, and cost. Treat the tokenizer as a versioned dependency of the dataset, not as an interchangeable formatting tool.