01.03 · Concept
The Index: Your Page Is Not Stored as a Page
Explain tokenisation and the inverted index, and say what actually gets looked up when someone searches - a posting list, not your document.
Search lookup starts with tokens and posting lists, not by rereading your page. A crawler-processed document is normalised into searchable units, and the index maps each unit to matching documents plus metadata. Queries are tokenised too, then their lists are combined to produce candidates for ranking.
What this lesson answers
- what is an inverted index in search
- does Google search my page at query time
- how does tokenisation affect search results
Notes
A search engine does not usually answer a query by opening every saved web page and scanning it from top to bottom. Before search time, it breaks documents into tokens: normalized pieces of text such as words, stems, or other searchable units. For each token, it records which documents contain it, often with extra facts like positions, frequency, and fields. That structure is the inverted index.
The mental model is the index at the back of a book, but at machine scale. Instead of storing “page 57 contains cats,” it stores “cats appears in document 12, document 91, document 203...” When someone…
References
Common questions
- What does a search engine actually look up when someone searches?
- It looks up index entries for the query tokens. Each entry points to a posting list: the documents where that token appears, often with useful metadata such as location, frequency, or field. The engine combines those lists to build candidate results before ranking and displaying document data.
- Is my web page stored as a whole page in the search index?
- Not in the way people often imagine. Search systems may store fetched page data elsewhere, but the searchable structure is built from processed tokens and metadata. For search visibility, the important question is which terms, fields, and signals from the page were extracted into the index.
- Why does tokenisation matter for SEO?
- Tokenisation decides what text units the search system can match. Case handling, stemming, punctuation, language rules, and field extraction can all change which tokens are indexed. If important meaning is lost or split poorly during processing, relevant queries may not connect cleanly to the page.
Short definition: what is Index: Your Page Is Not Stored as a Page?