#2 Event-Driven Architecture
Distributed Stream (Kafka-like)
Build a durable, ordered, replayable log that many producers append to and many consumers read at their own pace.
What is being asked
Design a distributed log like Kafka: producers append messages to named topics, the messages are kept on disk for a retention period, and independent groups of consumers read them in order and can re-read from any earlier point.
Requirements
Functional
- Append-only topics, split into partitions; order is guaranteed within a partition.
- Consumer groups: each message goes to one consumer in a group, and every group sees every message.
- Replay from a chosen offset; retention by time or size.
Non-functional
- Millions of messages per second.
- No loss of acknowledged messages when a broker dies.
- Throughput that scales by adding brokers and partitions.
Design
- A topic is divided into partitions. Each partition is an append-only file, split into segments, stored on one broker (the leader) and copied to followers.
- A producer picks a partition, usually by hashing a key so related messages keep their order, and sends batches to that partition's leader.
- The leader writes to its log and followers pull the new records. A write is acknowledged once the in-sync replicas have it.
- Consumers pull by offset. The broker only stores each group's committed offset, so consumers are cheap and can rewind.
- A controller tracks which broker leads each partition and elects a new leader from the in-sync replicas on failure.
Key decisions and trade-offs
- Sequential disk writes and the page cache are why a log is fast: no random I/O, and data is sent to consumers straight from the cache.
- Acknowledgement level. Waiting for all in-sync replicas is safe but slower; leader-only acknowledgement is faster and can lose data on failover.
- Partition count. More partitions give more parallelism but more files, longer failovers and weaker ordering across keys.
- Delivery guarantees. At-least-once is the default; exactly-once needs idempotent producers and transactional offset commits.
Follow-up questions
- What happens when a consumer in a group crashes mid-batch?
- How do you handle one key that receives far more traffic than the others?
- How would you delete a single user's messages from an append-only log?
- How does a follower that was down for an hour catch up?