Prompt caching
Prompt caching is a provider-side optimisation where repeated leading tokens in LLM requests are processed once and reused on later calls. It works best when large, stable prompt prefixes such as instructions, tool definitions, schemas, examples, or policy text appear before the user-specific or changing parts of the request.
The problem is that many LLM applications resend the same bulky context on every call. Agent instructions, tool schemas, formatting rules, product background, and examples may dwarf the actual user request. Without caching, the provider must run the model over that repeated prefix each time, adding latency and input-token cost even though nothing important has changed in that part of the prompt.
Prompt caching works by recognising an identical prefix at the start of a request and reusing internal computation for it. The useful prompt shape is stable material first, volatile material last. Put system rules, fixed context, output schemas, tool definitions, and few-shot examples before the current user message, retrieved documents, timestamps, request identifiers, or conversation-specific text. Deterministic ordering matters because a small edit near the top changes the prefix.
The tradeoff is that you must treat prompt layout like an interface, not a scratchpad. Moving dynamic data earlier can make the prompt read naturally to a human while destroying cacheability. Caching also does not replace application state or memory: the model still needs the relevant information in the request. It only avoids recomputing repeated leading tokens; it does not remember omitted facts or infer that two differently written prefixes mean the same thing.
Engineers meet prompt caching when building agents, chat systems, code assistants, document workflows, or any product with long common instructions. It affects prompt templates, middleware, retrieval placement, tool registration, and observability. When costs or latency look high, inspect whether the request has a large shared prefix, whether it is byte-for-byte or token-for-token stable enough for the provider, and whether volatile fields are accidentally placed before reusable content.
Common questions
- Is prompt caching the same as conversation memory?
- No. Conversation memory is application or model behaviour that preserves information across turns. Prompt caching is an execution optimisation for repeated prompt prefixes. The request still has to include whatever the model should reason over. If context is missing, caching will not recover it; it only makes repeated leading material cheaper or faster to process.
- What should go at the start of a cached prompt?
- Put content that is large, reused, and deterministic at the beginning: system instructions, safety or policy text, tool definitions, output schemas, fixed product context, and examples. Put changing content later: user messages, retrieved snippets, request identifiers, timestamps, user profile details, and anything whose order or wording may vary between calls.
- Why did my prompt caching not help much?
- Usually the reusable prefix is not actually stable, is too small to matter, or is interrupted by dynamic fields. A timestamp, randomised tool order, personalised sentence, or retrieved passage near the top can make each prefix different. The honest answer depends on the provider’s cache rules, but prompt structure and deterministic rendering are the first things to check.