Dapper, a Large-Scale Distributed Systems Tracing Infrastructure
Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, et al.2010Google Research 2010
Read it on research.google(opens in a new tab)Why this one
Read this after The Tail at Scale and before you wire OpenTelemetry into a service mesh. The central idea is not "logs, but distributed." It is that a request needs a causal skeleton, a trace id plus spans, carried through every hop cheaply enough that production systems will tolerate it. What people get wrong is treating tracing as a dashboard feature you add after the architecture is done. Dapper shows the opposite: observability has to ride inside RPC boundaries, sampling policy, propagation formats, and developer ergonomics. The paper is worth your evening because it explains why modern traces look the way they do, and why they fail when one library drops context or one team invents its own ids. If you build services, queues, or agents that call tools, this is the paper that turns "latency is high" into a path you can actually inspect.
What to take away
- A trace id is useful only if every process preserves it across RPC, queues, and retries.
- Sampling must be cheap, centralized enough to reason about, and safe to run in production.
- Spans turn a slow request from a metric into a causal tree of service calls.
Reads with
- The Tail at Scale
traces turn its fanout latency problem into the slow path you can actually inspect
- Time, Clocks, and the Ordering of Events in a Distributed System
dapper's span tree is a production version of causality carried through messages