06.12 · Concept
Self-Host or API
Decide between self-hosting and a hosted API on measured numbers: break-even volume, latency requirements, data residency and the engineering time a fleet actually costs. Accept that the API often wins.
Hosted APIs usually win until measured traffic, latency constraints, data residency, or model control justify owning an inference fleet. The decision is a cost and operations comparison: token price versus GPU utilisation, engineering time, replicas, observability, incidents, queueing behaviour, and compliance work.
What this lesson answers
- when is self hosting inference cheaper than an API
- how to calculate inference API break even volume
- what hidden costs come with running GPU inference
Notes
The decision mechanism is a total-cost and constraint comparison between buying tokens from an API and operating an inference fleet yourself. For a fixed model class, self-hosting wins only when while satisfying latency, residency, reliability, and staffing constraints. A useful first-order formula is , versus .
Common questions
- How should I decide between self-hosting inference and using an API?
- Compare the full monthly cost of each option under your real workload. For self-hosting, include GPUs, idle capacity, replicas, networking, observability, upgrades, incidents, and named engineering ownership. For an API, include input and output token pricing plus contractual requirements. If the API meets latency and residency needs, it often remains the cheaper choice.
- Does higher GPU throughput automatically make self-hosting cheaper?
- No. Throughput only helps when the fleet is kept busy without breaking latency targets. Continuous batching, prefix caching, faster attention kernels, and better serving runtimes can improve utilisation, but they do not remove fixed cost. A lightly loaded private cluster can still cost more than an API because most of the paid capacity sits idle.
- When is self-hosting required even if it costs more?
- Self-hosting or a private managed deployment is justified when constraints are hard rather than preferential. Examples include regulated data that cannot leave a region or tenant, air-gapped environments, custom weights, unusual tokenisation, specialised decoding rules, or kernel-level serving changes. In those cases, price is secondary to control, compliance, and product requirements.
Short definition: what is Self-Host or API?