Document Ingestion and Chunking
Document ingestion and chunking is the pipeline step that turns raw files, pages, tickets, or wiki entries into cleaned text records, split into retrievable pieces with source metadata. It prepares content for embedding by extracting useful text, preserving provenance, and choosing chunk boundaries that balance semantic context with search precision.
RAG systems need this step because source material is messy and retrieval works on records, not on arbitrary files. A PDF may contain headers, footers, tables, and page artefacts; an internal page may mix navigation with content; a ticket may include repeated quoted replies. If that material is embedded as-is, the vector store receives noisy, poorly attributable text, and the answer layer cannot reliably cite or refresh its sources.
A typical ingestion job reads from each source, extracts the main text, normalises it, then splits it into chunks. Splitting may use simple character or token windows, overlap between neighbouring chunks, or document structure such as headings, paragraphs, pages, and sections. Each chunk becomes a record containing the text to embed plus metadata such as source URI, document identifier, title, position, timestamp, permissions, and domain labels.
The hard part is that there is no universally correct chunk size. Large chunks preserve surrounding context but can retrieve broad, unfocused passages. Small chunks match precise questions better but may omit the information needed to interpret them. Overlap reduces boundary loss but increases storage, embedding work, and duplicate-looking results. Structure-aware chunking is often cleaner, but it depends on source documents having reliable structure to begin with.
Engineers meet ingestion and chunking when building the offline or asynchronous side of a RAG pipeline. It is usually a batch job, queue worker, or scheduled sync that feeds an embedding service and vector database. Good implementations are repeatable and idempotent: reprocessing the same document updates or replaces the right chunk records rather than creating duplicates, and metadata makes filtering, debugging, citations, and permission checks possible.
Common questions
- Is chunking just splitting text every few characters?
- That is one possible strategy, but it is usually the least informed one. Effective chunking tries to keep related ideas together while making each record narrow enough to retrieve for a specific query. Headings, paragraphs, pages, sections, and modest overlap can all produce chunks that are more meaningful than blind fixed-width slices.
- Why not embed the whole document as one record?
- Whole-document embeddings often blur many topics into one vector, so retrieval may return a document for the wrong reason or miss the relevant passage entirely. Even when a document fits within model limits, it can contain unrelated sections. Chunking gives the retriever smaller semantic targets and gives the answer generator more focused context.
- What metadata should every chunk carry?
- At minimum, a chunk should identify where it came from and where it sits in the source. Useful fields include a source URI, stable document identifier, title, page or section marker, chunk order, timestamp, and access permissions. Extra domain labels can support filtering, but provenance and permissions are the fields teams regret omitting.
- How do I choose the right chunking strategy?
- It depends on document shape, query style, and retrieval behaviour. Structured manuals often benefit from heading or section-based chunks. Short support tickets may need little splitting. Long prose may need overlapping windows. The practical test is whether retrieved chunks contain enough context to answer questions without dragging in unrelated material.