What is the tool schema tax?

It is the inference cost of putting callable tool definitions into the model input before generation starts. The model must prefill those schema tokens even if no tool is chosen. That cost shows up as extra time to first token unless the exact schema text is already available through prefix cache reuse.

Why does prefix placement matter for tool schemas?

Prefix caches reuse the beginning of a prompt when the bytes match a previous request. If tool definitions come after chat history, retrieved context, or user text, the shared part is no longer at the start. Put fixed instructions and schemas first, then append request-specific content, so cache hits are possible.

When does caching tool schemas fail to help?

It fails when the schema block changes per request, contains tenant-specific values, timestamps, scopes, or generated prose, or is placed after variable text. It can also be a poor trade when many rarely reused tool sets occupy cache space that would otherwise hold hotter prefixes.

Serving Agents

07.07 · Walkthrough

The Tool Schema Tax

Measure what a tool definition costs before any tool is called: a handful of schemas adds hundreds of tokens to the prompt of every single request, which is real time-to-first-token unless that block is prefix-cached. Then show the fix: put the schemas where the cache can hold them.

Tool schemas add prompt tokens before any tool runs, so they increase prefill work and time to first token on every request. The practical fix is to make the tool block a stable prefix, before user-specific text, so the serving stack can reuse cached KV instead of recomputing the same schema.

What this lesson answers

why tool schemas increase time to first token
how to measure tool schema prompt overhead
where to place tool schemas for prefix caching

Notes

The tool schema tax is the prefill work induced by serializing callable tool definitions into the model’s input before any tool is called. If a request has user tokens $U$ , conversation tokens $C$ , and tool-schema tokens $S$ , the first forward pass processes $N = C + U + S$ tokens, and the avoidable part is $S$ unless the exact schema prefix is served from a prefix cache.

References

Tool-calling benchmarks and latency

Common questions

What is the tool schema tax?: It is the inference cost of putting callable tool definitions into the model input before generation starts. The model must prefill those schema tokens even if no tool is chosen. That cost shows up as extra time to first token unless the exact schema text is already available through prefix cache reuse.
Why does prefix placement matter for tool schemas?: Prefix caches reuse the beginning of a prompt when the bytes match a previous request. If tool definitions come after chat history, retrieved context, or user text, the shared part is no longer at the start. Put fixed instructions and schemas first, then append request-specific content, so cache hits are possible.
When does caching tool schemas fail to help?: It fails when the schema block changes per request, contains tenant-specific values, timestamps, scopes, or generated prose, or is placed after variable text. It can also be a poor trade when many rarely reused tool sets occupy cache space that would otherwise hold hotter prefixes.

Short definition: what is Tool Schema Tax?

07.06 · Serving AgentsTool Calls in the Loop 07.08 · Serving AgentsStructured Output