Memory that is not a vector database
Memory that is not a vector database is agent state stored in forms chosen for the access pattern: scratchpads for working notes, files for durable artefacts, databases for structured source-of-truth records, and other stores where semantic similarity search is not the main operation.
The problem is that agent memory is often treated as if every future need were retrieval of similar text. Most useful state is not like that. An agent may need its current plan, the exact user preference, a generated report, an audit record, or a permission check. Similarity search can surface related material, but it does not by itself provide identity, freshness, ordering, authorisation, or consistency.
The concrete choice starts with the later operation. A scratchpad is transient working state passed between steps: notes, hypotheses, tool results, and next actions. A file stores a durable blob or human-readable artefact, such as code, logs, transcripts, or reports. A database stores records with keys, constraints, updates, and permissions. Embeddings are appropriate when the operation is fuzzy recall over unstructured text and no exact key or schema is available.
The trade-off is that no single store gives every property cheaply. Scratchpads are simple but ephemeral and easy to pollute. Files are inspectable and portable, but poor at enforcing relationships or partial updates. Databases require schema and operational discipline, but give reliable identity and mutation. Embedding retrieval is flexible, but can return stale, duplicated, merely related, or unauthorised content unless guarded by metadata and source-of-truth checks.
Engineers meet this decision when designing agent loops, tool interfaces, and persistence layers. Put the agent’s immediate reasoning in a scratchpad, write produced artefacts to files, store canonical application facts in a database, and index messy text only when semantic search is genuinely needed. In practice, robust systems combine these stores rather than forcing all memory through embeddings.
Common questions
- Why not put all agent memory in a vector database?
- Because a vector database answers a narrow question: what text is semantically close to this query. Many memory reads need exactness instead, such as the current plan, the latest account state, a file path, or a permission. Similarity is useful for recall, but it is not a substitute for truth, identity, or consistency.
- When is a scratchpad the right memory?
- Use a scratchpad for temporary state inside an agent task: intermediate observations, reasoning notes, selected tool outputs, open questions, and next steps. It is best when the state should guide the current run but does not need to become a durable record or a reliable fact for future users.
- When should agent memory be a database instead?
- Use a database when the remembered item has identity, structure, updates, constraints, or access rules. User preferences, orders, tickets, permissions, billing state, and audit trails should not be recovered by semantic similarity. The database should remain the source of truth, even if its text is also indexed for search.