Distributed systems · 4 min read
What I Learned About Kafka.
A practical introduction to Kafka: how events move through topics and partitions, how consumers keep their place, and why teams use an event log.
Each consumer group tracks its own progress through the same event stream.
Thinking in events
Kafka is a system for moving and storing streams of events. An event is a small record that something happened: an order was placed, a payment cleared, or a user signed in. An application that creates events is called a producer; an application that reads them is a consumer.
The producer writes an event to Kafka, so it does not need to know which other services will use it. Consumers can read the event at their own pace. This loose connection is useful when several parts of a system need the same information, or when one service is temporarily unavailable.
For example, an online store can publish an `OrderPlaced` event once. The payment service can process it, the inventory service can reserve stock, and an analytics service can count it. The order service does not need a direct connection to every consumer, so adding another use for the event does not require redesigning the original producer.
Topics, partitions, and offsets
A topic is a named stream of related events, such as `orders`. Kafka splits a topic into partitions so it can store and process more data across multiple machines. Within one partition, events have a stable order. If the order of events matters for a particular entity, such as one customer's account, those events can use the same key so they are routed to the same partition.
Kafka keeps events for a configured period instead of removing each one as soon as it is read. A consumer tracks its position with an offset, which is like a bookmark in a partition. If it needs to recover from a failure, it can continue from its last saved offset or replay earlier events.
Partitions also set a useful limit: order is guaranteed within one partition, not across every partition in a topic. A message key, such as an order ID or customer ID, can keep related events together in one partition. That gives you per-key ordering while still letting unrelated keys be processed in parallel.
What clicked for me
A consumer group lets multiple copies of a service share the work: Kafka assigns partitions across the group so each partition is handled by one member at a time. A different consumer group can independently read the same topic for a separate job, like updating a dashboard while another service sends receipts.
If a topic has four partitions, a group can use up to four consumers to process them in parallel; extra consumers in that same group will wait until a partition becomes available. This is one way Kafka scales work, while keeping each partition's order intact.
My key takeaway is that Kafka is not just a pipe that delivers a message and forgets it. It is a durable event log that lets independent applications process the same history. Consumers can replay old events, so handlers should be safe to retry: for example, recording that an order was paid should not charge the customer twice if the same event is seen again.
When Kafka is a good fit
Kafka is helpful when many services need the same stream, event volume is high, or downstream work should continue even when a consumer is briefly offline. It also gives teams room to add new consumers without changing the producer.
It does add operational and design work. A small application with one simple request-response flow may be easier to build without Kafka. When choosing it, I would ask whether replay, independent consumers, throughput, or loose coupling solve a real problem for the system.