#48 XMPP/WebSocket Gateways

WhatsApp Messaging

Deliver messages between two billion users in real time, in order, with delivery receipts, and store them only until they are delivered.

Asked at Meta

What is being asked

Design one-to-one and group chat: a message sent from one phone appears on the other within a moment if it is online, or when it next connects, exactly once and in order.

Requirements

Functional

One-to-one and group messages; sent, delivered and read receipts.
Offline delivery; online presence and last seen; media attachments.

Non-functional

Hundreds of millions of concurrent connections.
No lost messages; no visible duplicates; order within a conversation.
End-to-end encryption: servers cannot read content.

Design

Boxes are services; rounded shapes are data stores and queues.

Each phone keeps one persistent connection to a gateway. A registry records which gateway currently holds each user.
The sender's message, with a client-generated id, goes to the router, which writes it to the recipient's inbox queue and acknowledges to the sender (one tick).
If the recipient is connected, the router forwards the message to their gateway. The recipient's phone acknowledges, the message is removed from the inbox, and a delivery receipt goes back (two ticks).
If the recipient is offline, the message waits in the inbox and a push notification wakes the phone, which connects and drains its inbox.
Media is uploaded separately, encrypted; the message carries only a reference and key.

Key decisions and trade-offs

Store until delivered. Keeping messages on the server only while undelivered limits storage and exposure; history then lives on the devices.
Idempotent by message id. Retries are common on mobile networks, so the recipient de-duplicates by id.
Ordering is per conversation, using sequence numbers assigned for that conversation rather than device clocks.
Group messages are fanned out to each member's inbox. With end-to-end encryption a shared sender key avoids encrypting separately for every member on each message.

Follow-up questions

How do you handle one user signed in on a phone and a laptop?
How would you implement "last seen" for a billion users cheaply?
What happens when a gateway holding a million connections dies?
How do new group members get keys without seeing old messages?