Scoring Text: TF-IDF and BM25
TF-IDF and BM25 are lexical ranking formulas that score how well a document matches a query by combining term occurrence, term rarity, repetition limits, and document length. TF-IDF gives more weight to frequent terms in a document and rare terms across the corpus; BM25 refines that idea with saturation and length normalisation.
The basic problem is that a search engine cannot treat every word match as equally useful. A match on a rare technical term usually says more about relevance than a match on a common word. At the same time, simply counting occurrences is unsafe: a very long page, or a page stuffed with repeated terms, can look artificially relevant even when it is not the best answer.
TF-IDF handles this by multiplying two ideas. Term frequency rewards a document for using the query term more often. Inverse document frequency rewards terms that appear in fewer documents, because they distinguish documents better. BM25 keeps that lexical voting model, but changes the term-frequency part so repeated mentions add less value after the early occurrences, rather than growing without restraint.
BM25 also adjusts for document length. It compares a document with the average document length in the collection, so the same number of matches can mean different things in a short focused document and a large general one. This is the point of length normalisation: not to punish useful long documents, but to stop length alone from acting like relevance.
The trade-off is that BM25 is still not semantic understanding. It scores terms, not concepts, unless the indexing or query pipeline expands, rewrites, stems, or otherwise connects related words. Its behaviour also depends on tokenisation, corpus composition, field weighting, and parameter choices. The honest answer to “which document wins?” is: compute the term rarity, repetition effect, and length adjustment.
Engineers meet TF-IDF and BM25 in search engines, database text search, retrieval systems, and ranking explanations. They are useful when you need a fast, inspectable baseline for keyword matching. If one result beats another, you can usually trace it to rarer matched terms, more useful early repetitions, a shorter or more focused field, or a combination of those signals.
Common questions
- Does BM25 understand synonyms or intent?
- No. BM25 is a lexical scorer: it rewards matching terms according to rarity, repetition, and length. It does not know that two different words mean similar things unless another part of the system adds that relationship through query expansion, stemming, synonyms, embeddings, or other preprocessing.
- Why is BM25 usually preferred over plain TF-IDF?
- Plain TF-IDF can over-reward repeated terms and long documents. BM25 keeps the useful idea that rare query terms matter more, but adds saturation so extra repetitions eventually help very little, and length normalisation so large documents do not win merely because they contain more opportunities to match.
- Can a shorter document outrank a longer one with the same query term?
- Yes. If both documents contain the same term count, BM25 may score the shorter document higher because the term is more concentrated. The length-normalisation component treats a match buried in a large document differently from the same match in a focused one, relative to the collection’s average length.