Tool Schema Tax
Tool Schema Tax is the extra inference work caused by including tool or function definitions in an LLM prompt before the model has called any tool. Those schemas are ordinary input tokens, so they add prefill latency to every request unless their exact prompt prefix can be reused from cache.
Agents often advertise their available tools by sending names, descriptions, parameter shapes, enums, required fields, and formatting instructions alongside each request. That feels like metadata, but to the model it is just more prompt. The cost appears before the first generated token and before any external API runs. This is commonly misunderstood: slow tool-using agents are not always slow because tools execute slowly. They may already have paid for reading the menu.
Mechanically, the serving engine tokenises the serialised tool definitions and runs them through the model during prefill with the rest of the prompt. In a decoder-only transformer, this creates the same per-layer key and value tensors as any other prefix text. If the tool block is the first stable bytes of the prompt, a prefix cache can store those tensors and later requests can start from the cached state instead of recomputing the schemas.
The trade-off is that cached prefixes occupy KV cache memory and only help when the prefix matches exactly. Changing tenant-specific fields, timestamps, locale-specific wording, auth scopes, or generated documentation inside the tool block can destroy reuse. Cache lookup, routing, and page management also have overhead, so tiny schemas may not be worth special handling. The honest answer is workload-dependent: measure cold, warm, and deliberately uncacheable layouts.
Engineers meet this when wiring tool calling into OpenAI-style APIs, vLLM, SGLang, TensorRT-LLM, or distributed inference stacks with prompt caching. The practical fix is prompt layout: put the system instruction, fixed tool schemas, and fixed output contract first, then append chat history, retrieved context, and the current user turn. If the second identical-schema request has much lower time-to-first-token, the tax is being amortised.
Common questions
- Is Tool Schema Tax the cost of executing tools?
- No. It is paid before tool execution and even before the model chooses whether to call a tool. The schemas are serialised into the prompt, tokenised, and processed during prefill. Actual tool latency is a separate cost that happens later, after the model emits a tool call.
- Why does putting schemas first matter?
- Most prefix caches reuse only identical leading prompt bytes. If the user message or chat history comes before the schemas, then two requests with the same tools still have different prefixes. Placing stable schemas first lets the server reuse their cached key-value tensors across requests.
- Can I remove the tax by making schemas shorter?
- Shorter schemas reduce the amount of prefill work, but they do not change the mechanism. The larger win is often making the schema block stable and cacheable. Do both when possible: remove verbose descriptions and redundant JSON, then place the remaining fixed schema where prefix caching can hit.
- When does prefix caching not help?
- It does not help when the schema text changes per request, when schemas are placed after user-specific content, or when traffic is spread to workers that do not hold the cached prefix. It can also be a poor trade when cache memory is scarce or the schema block is very small.