Deduplication and Contamination Control
Deduplication and contamination control are dataset hygiene practices for removing repeated training examples and preventing evaluation material from leaking into training. They combine exact matching, approximate text matching, and semantic similarity search so models learn from genuinely distinct data and evaluations measure generalisation rather than memorised examples.
Large training corpora are often assembled from crawls, repositories, exports, and user generated sources that overlap heavily. The same article may be mirrored, a code file may appear in forks, or an instruction answer may be copied with small edits. If those repeats remain, training compute is spent reinforcing artefacts rather than broadening coverage. Worse, if held out questions or answers are present in training, benchmark scores can look strong while measuring recall of leaked data.
Exact deduplication starts by canonicalising records, then comparing fingerprints. Typical normalisation includes case handling, whitespace cleanup, Unicode normalisation, and splitting at document or line level before hashing. Fuzzy methods break text into shingles or n-grams, represent overlap compactly, and use techniques such as MinHash, SimHash, and locality-sensitive hashing to retrieve likely matches without checking every pair. Embedding deduplication maps chunks into vector space and groups near neighbours whose meaning is substantially the same.
The hard part is choosing what counts as duplicate. Strict rules miss lightly edited copies; loose rules can delete legitimately different examples, minority phrasing, or useful variations of the same task. Embedding approaches add model and index cost, and their similarity scores are not universal truth. Thresholds depend on domain, chunking, language, and whether the goal is removing boilerplate, preserving diversity, or protecting an evaluation set.
Engineers usually meet this in data pipelines before pretraining, fine tuning, benchmark construction, or dataset release. A practical workflow runs exact hashes early, fuzzy matching on likely high overlap sources, and embedding search where semantic repetition matters. For contamination checks, evaluation items are treated as protected data: scan training candidates against them using exact, fuzzy, and semantic methods, record removals, and version both the evaluation set and filtering code.
Common questions
- Is deduplication just deleting identical files?
- No. Identical file removal is only the cheapest layer. Real datasets contain copied pages with changed headers, code with small edits, rewritten answers, and translated or templated content. Good deduplication uses a cascade: exact hashes for obvious repeats, fuzzy text overlap for near copies, and embeddings when semantic equivalence matters.
- How do I know whether a similarity threshold is correct?
- There is no universal threshold. It depends on record length, domain, chunking, language, and the cost of false removals versus missed duplicates. Teams usually sample candidate pairs around several thresholds, inspect them manually, and choose rules that match the purpose of the dataset rather than treating the score as objective.
- What is evaluation set contamination?
- Evaluation set contamination occurs when examples used for testing, validation, benchmarks, expected answers, or associated artefacts appear in the training data. The model may then produce the right answer because it has seen the item before, not because it has learned the underlying capability the evaluation was meant to measure.
- Can contamination be removed after training?
- Usually not cleanly. You can rerun evaluation with affected items removed, or train again on a filtered corpus, but you cannot reliably subtract memorised examples from an already trained model. That is why evaluation sets should be versioned, access controlled, scanned against training data, and audited before major training runs.