11.02 · Concept
Prompt caching
Structure a prompt so the stable half is cached and the bill drops.
Prompt caching works when repeated requests share the same leading prompt text, letting the provider reuse processing for that stable prefix. Put invariant instructions, schemas, examples, tool definitions and product context first, keep them deterministic, and move user-specific or volatile content to the end so cost and latency fall.
What this lesson answers
- how should I structure prompts for caching
- what breaks prompt caching in LLM requests
- does prompt caching store conversation state
Notes
Prompt caching is a way for an LLM provider to avoid reprocessing the same beginning of a prompt over and over. If many requests share a large stable prefix, the service can reuse internal work for that prefix, then only process the new suffix. For an application, that usually means lower input-token cost and lower latency, especially when the repeated part is long: system instructions, policies, tool schemas, examples, documentation, or a fixed coding standard.
The practical mental model is “put the common, boring, reusable material first; put the changing user-specific material last.” A good…
Common questions
- What is prompt caching in an LLM API?
- Prompt caching is provider-side reuse of work for the unchanged beginning of a prompt. When many calls start with identical instructions, schemas, examples or tool definitions, the service can avoid repeating all processing for that prefix and focus on the new suffix. It is mainly useful for long, stable prompt templates.
- How do I make a prompt cache-friendly?
- Put the stable material first: system rules, product context, output schema, tool definitions and fixed examples. Put changing content later, such as the user request, retrieved passages, conversation updates, timestamps or request identifiers. Keep the stable section text and ordering deterministic across calls.
- Is prompt caching the same as model memory?
- No. Prompt caching does not make the model remember missing context or previous calls. The request still needs to include everything the model should use. Caching is closer to reusing internal computation for identical leading tokens than storing semantic application state.
Short definition: what is Prompt caching?