05.02 · Walkthrough
Kafka Producers, Consumers, and Topics
Create a Kafka topic, publish structured events, consume them, and inspect offsets and consumer group behavior.
Kafka topics are partitioned event logs written by producers and read by consumers. Keys influence partition placement and ordering, while offsets record each consumer group’s progress independently. Understanding these mechanics lets you publish structured events, replay streams, scale consumers, and reason about duplicate processing after failures or rebalances.
What this lesson answers
- how do Kafka producers choose topic partitions
- what are Kafka offsets and consumer groups
- how can Kafka consumers replay old messages
Notes
Kafka is a distributed log used to move event data between systems in near real time. A topic is the named stream of events, and each event is an immutable record with a key, value, timestamp, and optional headers. Producers write events to topics, usually serializing structured data as JSON, Avro, or Protobuf. The event key matters because Kafka uses it to choose a partition, which gives ordering guarantees for records with the same key while allowing the topic to scale across partitions.
Common questions
- What is a Kafka topic?
- A Kafka topic is a named stream of immutable event records. Producers append records to it, and consumers read those records later. Internally, a topic is split into partitions, which let Kafka scale reads and writes while preserving order for records that land in the same partition.
- Why does the event key matter in Kafka?
- The key is used to decide which partition receives the record. Records with the same key normally go to the same partition, so consumers see them in order within that partition. A stable key is important when downstream code depends on per-entity ordering or consistent processing.
- Do Kafka consumers remove messages after reading them?
- No. Kafka keeps records according to retention settings, not according to whether a consumer has read them. Each consumer group stores its own committed offsets, which represent progress through each partition. That allows separate applications to read the same topic independently and replay data when needed.