#8 Async Messaging
Distributed Queue (RabbitMQ)
A message queue that hands each task to one worker, redelivers it if the worker dies, and sets aside messages that keep failing.
What is being asked
Design a queue service like RabbitMQ or SQS. Producers enqueue tasks; workers take them, process them and acknowledge. Unlike a log, a message is removed once it has been handled.
Requirements
Functional
- Enqueue, receive and acknowledge; a visibility timeout after which an unacknowledged message is redelivered.
- Delayed messages, retries with back-off, and a dead-letter queue.
- Optional ordering per group.
Non-functional
- No loss of accepted messages.
- At-least-once delivery.
- Thousands of queues, some idle and some very busy.
Design
- A front-end routes each queue to the broker group that owns it. The leader persists the message and replicates it before acknowledging the producer.
- A worker receives a message, which marks it invisible for a timeout rather than deleting it.
- When the worker acknowledges, the message is deleted. If the timeout passes first, the message becomes visible again and another worker gets it.
- A delivery count travels with the message. After a set number of attempts it is moved to a dead-letter queue for inspection, so one bad message cannot block the rest.
Key decisions and trade-offs
- At-least-once means duplicates. A worker may finish the job and crash before acknowledging. Consumers must be idempotent, usually by recording processed message ids.
- Ordering costs throughput. Strict order means one message in flight per group; most queues offer ordering only within a group key.
- Push or long-poll. Pushing gives lower latency but needs flow control (a prefetch limit); long polling is simpler and naturally paced by the worker.
- Queue or log. A queue fits tasks handled once by any worker. When several independent systems need the same events, or replay is needed, a log is the better tool.
Follow-up questions
- How do you choose the visibility timeout for jobs that take between one second and ten minutes?
- How would you implement a message that must be delivered in exactly one hour?
- What do you alert on to know that workers are falling behind?
- How do you drain and move a queue to another broker without losing messages?