09.02 · Walkthrough
Document Ingestion and Chunking
Build a document ingestion step that extracts text, chunks documents, attaches metadata, and prepares records for embedding.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Everything Data lesson by lesson; 55 of 85 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
Document ingestion turns raw files and pages into clean chunk records for embedding, while keeping enough source context to retrieve, filter, cite, debug and reprocess them safely. The key choices are text extraction, chunk boundaries, overlap, structure awareness, metadata design and idempotent handling of updates.
What this lesson answers
- how should I chunk documents for RAG
- what metadata should vector chunks include
- how to make document ingestion idempotent
Notes
Document ingestion is the first stage of a RAG pipeline where raw source material is converted into clean, structured records that can later be embedded and stored in a vector database. In practice, this means reading documents from sources such as PDFs, HTML pages, Markdown files, tickets, or internal wikis; extracting the useful text; and normalizing it enough that downstream systems can process it reliably.
Common questions
- Why not embed a whole document at once?
- Whole documents often mix separate topics, which makes similarity search less precise. Even when the embedding model accepts the input, a single vector can blur unrelated sections together. Smaller chunks give retrieval a better chance of matching the actual question, while still carrying enough surrounding text to preserve meaning.
- What makes a good chunk boundary?
- A good boundary keeps related text together and avoids cutting through an idea. Headings, paragraphs and sections are usually better signals than blind character windows. Some overlap is useful when meaning spans a boundary, but too much overlap can create noisy duplicates and increase storage and embedding work.
- What metadata should be stored with each chunk?
- Store enough metadata to trace, filter and maintain the record: source location, document identity, title, page or section, chunk position, timestamp, permissions and any domain labels your retrieval layer needs. The text is for embedding; the metadata is for provenance, access control, debugging, citation and clean reprocessing.
Short definition: what is Document Ingestion and Chunking?