01.04 · Lecture
Scoring Text: TF-IDF and BM25
Compute why one document scores higher than another for the same term, and explain what BM25's saturation and length-normalisation terms are correcting for.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through SEO lesson by lesson; 39 of 52 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
BM25 ranks lexical matches by combining term rarity, repeated occurrence, saturation, and document length. It preserves the TF-IDF idea that uncommon query terms carry more weight, while reducing the reward for stuffing the same term and correcting the bias that would otherwise favour long, unfocused documents.
What this lesson answers
- why does BM25 rank one document higher
- how does TF-IDF score search results
- what does BM25 length normalisation fix
Notes
Classic search scoring starts with a simple question: if the user searches for a word, which document should rank higher? TF-IDF answers by combining two signals. Term frequency says a document mentioning the term more often is probably more about it. Inverse document frequency says a rare term is more informative than a common one. A document matching “postgres” should get more credit than one matching “the,” because almost every document contains “the.”
A useful mental model is that each query term casts votes for documents. A rare term casts a stronger vote than a common term.
References
Common questions
- What problem does TF-IDF solve in search ranking?
- TF-IDF separates useful matches from weak ones by weighting terms according to both presence and rarity. A document gets more credit for matching a distinctive query term than for matching a word that appears almost everywhere. Repeated occurrences can increase the score, but plain TF-IDF does not fully handle repetition abuse or document length bias.
- What does BM25 saturation mean?
- Saturation means repeated mentions of the same query term have diminishing value. The first match strongly suggests relevance, and a few more can strengthen that signal. After that, additional repeats add much less. This stops a document from winning mainly because it repeats a keyword many times.
- Does BM25 understand synonyms or meaning?
- BM25 is a lexical scoring method, not a semantic model. It scores documents based on the terms that appear in the query and the indexed text. It will not treat related words as equivalent unless another part of the search system expands the query, rewrites terms, or indexes those relationships explicitly.
Short definition: what is Scoring Text: TF-IDF and BM25?