12.03 · Walkthrough
Deduplication and Contamination Control
Apply exact, fuzzy, and embedding-based deduplication techniques and check for evaluation set contamination.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Everything Data lesson by lesson; 55 of 85 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
Deduplication removes repeated, near-repeated, and semantically duplicated training examples so a model does not memorise artefacts, waste compute, or get inflated evaluation results. Contamination control applies the same matching ideas against held-out benchmarks and validation data, using hashes, overlap methods, embeddings, review, versioning, and audit trails.
What this lesson answers
- how to deduplicate LLM training data
- how to find benchmark contamination in training data
- when to use embedding based deduplication
Notes
Deduplication is the process of removing repeated or near-repeated examples from a dataset so the model does not overlearn common artifacts, waste training compute, or appear better than it really is. Exact deduplication handles byte-for-byte or normalized matches: for example, lowercasing, stripping whitespace, normalizing Unicode, and hashing documents or lines to detect identical records. This is cheap and should usually be done early.
Common questions
- What is the difference between exact and fuzzy deduplication?
- Exact deduplication finds records that match after normalisation, such as whitespace cleanup, case folding, Unicode handling, or hashing. Fuzzy deduplication finds records that are mostly the same but not identical, such as scraped pages with changed headers, lightly edited code, or republished articles with small formatting differences.
- When should embedding-based deduplication be used?
- Embedding-based deduplication is useful when duplicated meaning matters more than identical wording. It can identify paraphrases, templated responses, translated boilerplate, or repeated instruction patterns. It costs more than hash or overlap methods, and thresholds need review because aggressive filtering can remove legitimate variation from the dataset.
- How do you check for evaluation set contamination?
- Treat evaluation data as sensitive and compare it against training data before training. Use hashes for direct matches, fuzzy overlap for partial copies, and embedding search for paraphrased or reformatted examples. Keep versions clear and record removals so reported scores measure generalisation rather than memorisation.
Short definition: what is Deduplication and Contamination Control?