Skip to content
Klay

All papers

System Design18 min read

Building a Replicated Logging System with Apache Kafka

Guozhang Wang, Joel Koshy, Sriram Subramanian, et al.2015VLDB 2015

Read it on doi.org(opens in a new tab)
  • logs
  • streams
  • replication
  • kafka

Why this one

Read this after MapReduce and The Google File System, then before Dapper or any modern streaming architecture. Kafka's real idea is not a faster queue, it is treating an append-only, partitioned, replicated log as the shared spine of a company. Producers, consumers, replay, fanout, retention, and recovery all become simpler because the system refuses to hide ordering behind per-message RPC semantics. What people get wrong is calling Kafka pubsub and stopping there. The paper is about the operational shape that makes the abstraction survive: partition leadership, sequential disk IO, consumer-owned offsets, batching, and replication that is good enough to run at LinkedIn scale. If you build event pipelines, audit trails, async services, or materialized views, this is one of the papers that changes what you consider the database.

What to take away

  • Kafka makes consumer offsets external, so replay and independent fanout become normal operations, not recovery hacks.
  • Partitioned append-only logs trade global order for scalable per-partition order that applications can reason about.
  • The design leans on sequential disk writes and batching, proving durable messaging need not mean slow messaging.

Reads with