WhatsApp is a messaging app for one-to-one and group chats. A message shows up on the other person's phone well within a second when they are online, waits for them when they are not, and shows the sender one tick when the server has it, two when it is delivered and blue ticks when it is read. The server never sees what the message says.
The hard parts are holding hundreds of millions of open connections, reaching the one server that holds the recipient's connection, and never losing, duplicating or reordering a message. This write-up follows one message, "Landed! See you at 7", from Jane's phone to John's.
Now draw it yourself, cold
You can nod along to a finished design and still freeze five minutes into the real round. In the 14-day free trial you draw this system yourself while an interviewer asks questions and grades every decision.
Understanding the problem
Functional requirements
- Users can start one-to-one chats and group chats of up to 100 people.
- Users can send and receive text messages in real time.
- Senders can see when a message has been delivered and when it has been read.
- Users receive messages that were sent while they were offline, for up to 30 days.
- Users can send and receive photos and videos.
Questions worth asking, with the answers you will usually get:
- Can a user have several devices? Yes: a phone plus a few linked devices, such as a laptop. Every device gets every message.
- Does the server keep chat history? No. It holds a message until every device has it, or for 30 days, then deletes it. History lives on the devices.
- Is it end-to-end encrypted? Yes. The server routes ciphertext. You should know how that shapes the design; you do not need to derive the cryptography.
- How big can media be? Up to 100 MB.
- Are group sizes capped? At 100 for this problem. Real apps cap groups at around a thousand, and the cap matters later.
Out of scope: voice and video calls, typing indicators, online status and last seen, status updates, message search, backups and the internals of the encryption protocol. (If an interviewer adds presence, derive it from the connection registry, which already knows who is connected, rather than writing a "last seen" row on every heartbeat.)
Non-functional requirements
- Never lose an acknowledged message. Once Jane sees one tick, the message survives any single server failure.
- Fast delivery. An online recipient gets the message within 500 ms at p99.
- In order, once. Messages in a chat show in the same order on every device, with no duplicates.
- Scale. 1B users, 200M connected at any moment, 100B messages a day.
- Private and small. The server stores ciphertext only, and only until it is delivered.
The first requirement decides the most: the server writes a message to durable storage before it says "sent", and everything after that write can be best-effort, because the stored copy can always be delivered again.
Back-of-the-envelope estimation
Round hard, and keep the numbers that change a decision.
Messages.
messages: 100B / day ÷ 86,400 s ≈ 1.16M / s ≈ 1.2M / s
peak (2×): ≈ 2.3M / s
Deliveries. A message goes to every device of every other member, and to the sender's own other devices. Most chats are one-to-one, and groups pull the average up. Assume 3 devices per message.
deliveries: 1.16M × 3 ≈ 3.5M / s
= 3.5M inbox writes / s, and 3.5M deletes / s
receipts: 3.5M delivered / s + ~1M read / s
routed: 3.5M + 4.5M ≈ 8M events / s
Connections. A chat server holds open sockets that are idle almost all the time. With about 20 KB of buffers per socket, a 64 GB machine holds 500,000 of them comfortably.
200M sockets ÷ 500,000 = 400 servers at full load
3 zones, survive losing one: 400 × 3/2 = 600 servers
per server: 200M ÷ 600 ≈ 333,000 sockets
reconnects: ~1.5B devices × 20 a day ≈ 350,000 / s
Storage. A stored message is about 300 bytes: its ciphertext plus a few ids and timestamps.
written: 100B × 300 B = 30 TB / day
kept 30 days: 30 TB × 30 = 900 TB
until delivered: say 10% wait about a day for an offline device
100B × 10% × 300 B ≈ 3 TB
Media. Say 5% of messages carry a photo or video, 100 KB on average after compression.
files: 100B × 5% = 5B / day
bytes: 5B × 100 KB = 500 TB / day ≈ 5.8 GB / s ≈ 46 Gbit / s
text: 1.2M × 300 B ≈ 360 MB / s
| Quantity | Estimate | What it decides |
|---|---|---|
| Open sockets | 200M | A fleet of 600 chat servers, and a registry of who is where |
| Messages | 1.2M / s | A store that scales writes horizontally, keyed by chat and seq |
| Inbox writes and deletes | 3.5M / s each | Inbox rows are small pointers, not copies of the message |
| Routed events | ~8M / s | Publish to one server, never to all 600 |
| Storage if kept 30 days | 900 TB | Delete on delivery, which keeps about 3 TB |
| Media | 46 Gbit / s | Files go straight to S3, never through a chat server |
Core entities
- User, and Device: each of a user's phones and laptops, with its own public keys.
- Chat: one-to-one or group, with its members and
last_seq, the counter that numbers its messages. - Message: chat id,
seq, sender, time, and the ciphertext. - Inbox entry: one per device that has not yet received a message: device id, chat id, seq. It points at the message; it does not copy it.
- Connection: which chat server holds each online device's socket. This is in memory, not in the database.
A one-to-one chat is a group of two, so the rest of the design never has to treat it differently.
API
Messages flow both ways and the server has to push them, so each device keeps one WebSocket open. Everything about messages travels over it as small frames; only key lookup and media use HTTP.
GET /ws (Authorization: Bearer <device token>)
→ 101 Switching Protocols, or 401 for a bad token
→ createChat { members: ["john"], name: null }
← chatCreated { chatId: 9 }
→ sendMessage { chatId: 9, clientMsgId: "c-881", payload: <ciphertext> }
← ack { clientMsgId: "c-881", seq: 42 }
← error { clientMsgId: "c-881", code: "notMember" | "staleDevices" }
← newMessage { chatId: 9, seq: 42, sender: "jane", sentAt, payload }
→ delivered { chatId: 9, seq: 42 }
→ read { chatId: 9, upTo: 42 }
← receipt { chatId: 9, seq: 42, by: "john", state: "delivered" | "read" }
GET /keys/{userId} → 200 { devices: [ { deviceId, keys } ] }, or 404
POST /media → 201 { mediaId, uploadUrl, expiresIn: 300 }
Three choices to say out loud. The client names its message with clientMsgId, so a retry after
a dropped connection is recognised and not stored twice. The server numbers it: seq comes back
in the ack and is the order every device uses. delivered is the device's ack, and the server
deletes nothing until it arrives; that frame is what makes delivery reliable.
High-level design
Start with the smallest design that meets each requirement, on a single chat server where that is enough. Many servers, ordering, groups, devices and encryption come in the deep dives.
1. Users can start a chat
- Jane's phone opens a WebSocket through an L4 load balancer to a chat server. The load balancer works at the TCP level, so it passes a long-lived connection through without parsing it.
createChatwrites a Chat row and one member row per person to DynamoDB: keyed by chat id, with a secondary index on user id. The chat server can then find a chat's members, and a user's chats.- The server answers
chatCreatedwith the new id.
See it in the flythrough02 · One socket
2. Users can send and receive messages in real time
Walk Jane's message through it:
- Store. Server A checks that Jane is a member of chat 9, takes the next number from the chat's counter, 42, and writes the message. Then it writes an inbox entry for every device that should get it: John's phone, John's laptop and Jane's own laptop.
- Acknowledge. Only after both writes succeed does it send
ack { seq: 42 }. Jane sees one tick. - Push. For each recipient device connected to this server, it sends
newMessagedown that device's socket. - Delivered. John's phone answers
delivered. The server deletes that device's inbox entry and sends Jane a receipt: two ticks. - Read. When John opens the chat, his phone sends
read { upTo: 42 }. One frame covers every earlier message too. Jane's ticks turn blue.
See it in the flythrough03 · Stored first
Why write the inbox entry when John is online? Because the push can fail: the socket can be half
dead, or the server can crash after the ack. The inbox entry records what each device is still owed
until that device's delivered. Online delivery is the fast path; the inbox makes it correct.
See it in the flythrough04 · One tick
Receipts travel the same route as messages. They have no seq of their own, so if Jane is offline her device keeps one receipt entry per chat in its inbox, holding the latest delivered and read positions, overwritten rather than added to.
See it in the flythrough08 · Receipts
3. Users get messages sent while they were offline
- Store as usual. John's phone is in a tunnel. The send path does not change.
- Wake the phone. Finding no connection for John's devices, the server asks a push service to notify through APNs (iOS) or FCM (Android), since a backgrounded app cannot keep a socket open. The notification carries no text; the app decrypts on wake to show a preview.
- Drain on reconnect. When John's phone connects, its server queries the inbox for that device, sorted by chat and then seq, sends the messages a page at a time, and deletes each entry when the phone acks it.
- Expire. Every inbox entry and message has a 30-day TTL.
See it in the flythrough10 · John offline
See it in the flythrough11 · Drain the inbox
4. Users can send media
A 100 MB video would hold a chat server's socket and memory for minutes, so media goes around the chat path:
- Jane's phone encrypts the photo with a new random key and asks the media service for an
upload URL:
POST /mediareturns a presigned S3 URL that allows PUTs to one key for 5 minutes. A 100 MB video uses a multipart upload, so a dropped connection resumes from the last part. - The phone uploads the encrypted file straight to S3.
- It sends an ordinary message whose encrypted payload holds the S3 object key, the decryption key and a SHA-256 hash of the file, so the server never holds the key.
- John's phone downloads the file through a CDN, checks the hash and decrypts it.
S3 lifecycle rules delete files after 30 days. Every file is ciphertext, so a CDN can cache it safely for the 40 people in a group who download the same photo.
See it in the flythrough17 · Media
Deep dives
1. How does a message reach the server holding John's socket?
With 600 chat servers, Jane's phone is on server A and John's is on server B. Server A has the message; server B has the socket.
BadSend every message to every server
Every chat server subscribes to one shared channel, receives every message and keeps the ones for sockets it holds. It needs no registry and no record of who is where, so it is the first thing that works with two servers. At 600 servers it is 1.2M messages a second × 600 = 720M deliveries a second, and 599 in 600 are thrown away. At 300 bytes a message, each server takes in 360 MB a second, about 2.9 Gbit/s, and checks 1.2M messages a second against its socket table, to deliver about 2 MB a second. Each server added to hold more sockets adds another 1.2M deliveries a second, so growing the fleet makes it worse.
See it in the flythrough05 · Bad: every server
GoodA pub/sub channel per user
When John's phone connects, server B subscribes to user:john on Redis pub/sub, and server A
publishes there. Senders need no lookup, and only the server holding John's socket gets the message,
so it looks like the fix. At this size Redis holds 200M live subscriptions, and each of the 350,000
reconnects a second is an unsubscribe on the old server and a subscribe on the new one, 700,000
changes a second. Worse, a classic PUBLISH in Redis Cluster is copied to every node, so the 8M
events a second are broadcast to all of roughly 80 shards: 640M node deliveries a second, the Bad
option's problem moved into Redis.
GreatA connection registry and a channel per server
- Registry. When a device connects, its server writes one field to a Redis hash:
HSET conn:john phone B. Disconnecting deletes it. With 350,000 reconnects a second that is about 700,000 writes a second, and lookups add one read per recipient user, a few million a second, spread over a Redis cluster. - Look up, then publish. Server A reads
HGETALL conn:john, learns that the phone is on B and the laptop on C, and publishes the message once toserver:Band once toserver:C. Each server subscribes to exactly one channel, its own, with Redis 7's sharded pub/sub (SSUBSCRIBE,SPUBLISH), so a publish goes only to the shard that owns the channel. Sending straight to B over gRPC, since the registry gives its address, works as well. - Offline is free. A device with no entry gets no publish, and the push service is called instead.
See it in the flythrough06 · Great: look it up
The routing layer then carries the 8M events a second from the estimate. At about 100,000 publishes a second per Redis shard, that is roughly 80 shards, plus replicas.
What a senior candidate adds:
- Pub/sub is allowed to lose messages. It stores nothing, so a publish to a restarting server is gone. The message is already in the inbox, so the worst case is a delay, never a loss.
- A lost zone. The 600-server sizing assumes it: 200 servers and about 66M sockets drop at once. Clients reconnect with random backoff over about 2 minutes, about 550,000 connects a second on top of the normal 350,000, and servers refuse new connections above a safe rate so the rest retry later. Each device lands on another server and drains its inbox.
- Stale registry entries. A crashed server cannot clean up its entries. Expiring each entry would mean refreshing 200M of them every few minutes. Instead each server heartbeats once a second, and a lookup that names a dead server treats the device as offline. Each field also holds a connection id, and a disconnect deletes the field only if the id still matches, so a late delete from B cannot remove the entry the phone just wrote on D.
- A lost publish while connected. If a publish is lost and John stays connected, nothing on server B knows the message exists. Two things recover it. When the next message in the chat arrives, his phone sees seq jump from 57 to 59 and asks for a sync of chat 9, answered from the inbox; if 58 is not there, it is still on its way or was skipped. And in case no next message comes, every 5 minutes the phone's keepalive asks server B to check its inbox: 200M devices ÷ 300 s is about 670,000 small reads a second, almost all empty, and it bounds the delay at 5 minutes. The same keepalive, a ping every minute, is how both ends notice a half-dead mobile socket.
- Half-dead sockets. A phone that walks out of coverage leaves a socket that looks open. If
nothing is being sent, TCP never notices (its own keepalive defaults to two hours), and with data
in flight its retransmissions take many minutes to give up. So the
application pings. If server B hears nothing for two ping intervals, it closes the socket and
deletes the registry entry. And when B pushes a message and the device's
delivereddoes not come back within a few seconds, B pushes it again from the inbox entry, which is still there; the device drops the copy by seq if the first one did arrive. - Sizing is about memory, not bandwidth. Deliveries and receipts come to about 1.3 GB a second across 600 servers, about 2 MB a second each. Sockets are what fill a server.
See it in the flythrough18 · At scale
2. How do you never lose, duplicate or reorder a message?
The network will drop frames, deliver them twice and deliver them late. "Exactly once" is not something the network can give you; you build it from at-least-once delivery and duplicates that do no harm.
Never lose: store before acking, keep the inbox entry until the device acks, and retry from the inbox. Never duplicate:
- On the way in. Jane's phone retries
sendMessagewith the sameclientMsgIduntil it gets an ack. Server A takes a seq, then writes the claimdedupe:jane:c-881(holding that seq) and the message in one DynamoDB transaction, with the condition that the claim does not exist yet. Then it writes the inbox entries and acks. - A retry after a crash. A retry whose transaction fails on the condition reads the seq from the claim, writes the inbox entries again (plain puts, so a second write changes nothing) and acks with that seq. A crash between the transaction and the inbox writes therefore never leaves an acked message without its inbox entries, and a crash before the transaction leaves only an unused seq. The claim lives 30 days, as long as the message can, and the phone marks a send older than that as failed instead of retrying, so a late retry never creates a second copy.
- On the way out. A push whose
deliveredframe was lost is sent again on the next drain. The phone already has (chat 9, seq 42), so it drops the copy and acks again.
In order is the harder part. John asks "Where are you?" and Jane replies "Taxi now" a moment later. Every device must show them in that order.
BadSort by the sender's clock
Each phone stamps its message with its own time, and every device sorts by that time. It costs the server nothing, and the time is already in the payload for display. Phone clocks drift and are often set by hand, so they are wrong by seconds or minutes. John asks "Where are you?" at 18:43:05. Jane replies 5 seconds later, but her clock is 9 seconds slow, so her reply carries 18:43:01 and sorts above the question on every device. A phone whose date is a day ahead puts each message it sends below everything the chat writes for the next day.
See it in the flythrough12 · Bad: phone clocks
GoodSort by the server's clock
Stamp each message when the server receives it. Every device sorts by the same stored time, a phone's wrong clock stops mattering, and servers synced by NTP agree to within a few milliseconds, so this is right almost all the time. A few milliseconds is the normal case, not a guarantee. A server whose NTP sync stops working drifts at its crystal's rate, often 50 parts per million: about 4.3 seconds a day, 30 seconds after a week nobody noticed. John asks "Where are you?" through server A at 18:43:05. Jane replies 3 seconds later through server C, which stamps it 18:42:38, so her answer sits above the question on every device, and the stored order stays wrong after the clock is fixed.
GreatA sequence number per chat
The chat's row holds a counter. Storing a message increments it atomically:
UpdateItem chat 9
SET last_seq = last_seq + 1
RETURN UPDATED_NEW → 44
- One order per chat. Every device sorts chat 9 by seq, so all of them agree. Order across chats does not matter, so there is no global counter to bottleneck on.
- Cheap. A busy group sends a few messages a second; one DynamoDB partition takes up to 1,000 writes a second. Messages themselves are keyed by chat and seq, each its own item: the server never reads a chat's history, only single messages by pointer, so nothing else lands on the chat's partition.
- Gaps are allowed. If the counter moves and the message write then fails, that number is skipped. Seq is for ordering, not for counting, and the phone places any message by seq whenever it arrives.
- One send at a time. If Jane's phone sent two messages at once and the first timed out, its retry would get a later seq than the second. The phone sends a chat's messages one at a time, each after the previous ack.
See it in the flythrough13 · Great: a number per chat
3. How do group messages fan out?
Jane posts in "Goa trip", a group of 40 people with 2 devices each. There are two ways to get one message to 80 devices.
BadCopy the message into every inbox
Write the whole message into every member device's inbox: 80 items for one message in Goa trip, each with the full body. A device's drain is then one query that returns everything, with no second read, which is why it is tempting. With sender keys (deep dive 5) every member receives the same ciphertext, so 79 of the 80 bodies are the same bytes. At 3 deliveries a message, the copies triple what waits for offline devices, from about 3 TB to 9 TB. A full group of 100 members with 5 devices each writes the body 500 times. A message with a 4 KB caption is 4 write units per copy in DynamoDB, 2,000 for that one message instead of 500 for pointers.
GoodFan out at read time
Store each message once in the chat's log, and let each device keep a cursor per chat: the last seq it has. Sending is one write whatever the group's size, and nothing is copied. The cost moves to reconnecting. A device cannot know which chats have news without asking each one, and an active user is in hundreds of chats. At 350,000 reconnects a second and, say, 200 chats each, that is 70M cursor checks a second, nearly all answering "nothing new", where one inbox query per device would be 350,000. Online members still need a push for every message, so the server still looks up and publishes to every member, the fan-out this was meant to avoid.
GreatFan out at write time, to small pointers
Store the body once in Messages, and write a small inbox entry per member device: the device id, the chat id and the seq. Publish once per server that holds any of those devices, with the list of devices on it.
See it in the flythrough14 · Group fan-out
Write-time fan-out costs work proportional to the group's size on every message, and that is why the group cap matters. At 100 members and about 2 devices each, one message is about 200 inbox writes, and with the cap of 5 devices a user (deep dive 4) never more than 500: a burst, but a bounded one. At a cap of 1,000 members it is about 2,000, still fine if a fan-out job is stored before the ack and a background worker writes the rows beyond the first few hundred. Reconnecting stays a single query on one device's inbox, however many chats the user is in.
Read-time fan-out wins for a broadcast channel with a million mostly idle followers. Name the cap, and say the design would switch for channels.
4. How does every device get every message?
John has his phone and a laptop. Each needs every message, his own sent ones included, and each may be offline at different times.
- A device is the unit of delivery. Each device has its own socket, its own registry field and
its own inbox.
conn:johnholdsphone → Bandlaptop → C, and each device's inbox entry is deleted only by that device's ack. - Senders include their own devices. Jane's laptop gets her message through its own inbox entry, so the conversation looks the same there.
- Read state syncs. If John reads on the laptop, the
readframe goes to Jane and also, as a sync message, to his phone, so the phone clears its unread badge. - Cap the devices. A phone and four linked devices bounds the fan-out at 5 entries per user.
- A stale device list. If John links a new device after Jane's phone fetched his keys, her
payload has no ciphertext for it. Server A rejects the send with the current device list, and the
phone encrypts for the new device and retries with the same
clientMsgId. - Dormant devices. A laptop that stays shut keeps every message waiting for 30 days. Unlink a device after 30 days without connecting.
- The message is deleted when the last device acks. The Messages row carries a count of pending devices. An ack deletes the device's inbox entry with a condition that it still exists, and only a delete that removed something decrements the count, so a repeated ack for a duplicate push counts once. Zero deletes the row.
A new laptop has no history, because the server keeps none. The phone sends it recent chats directly, encrypted for the new device.
See it in the flythrough15 · Every device
5. How does end-to-end encryption shape the design?
The interviewer wants to know what the server can see, and what that costs.
BadTLS only
Encrypt each connection with TLS, and keep messages readable on the server. It is what most web apps do, it lets the server search, preview and filter spam, and it already stops anyone on the network from reading traffic. But the server decrypts every message to route it, so all 1.2M messages a second pass through the chat servers in plain text, and the 3 TB waiting for offline devices sits readable in the store. Anyone with access inside, whether an engineer, a stolen admin credential or a backup copied out, can read any chat. One breach of the message store exposes every undelivered message.
GoodEncrypt at rest with server keys
Encrypt the message store and S3 with keys the server holds, through a key management service, so the 3 TB of waiting messages are ciphertext on disk. DynamoDB and S3 do it by default, and it satisfies a compliance checklist. It protects against one thing: a disk or backup that leaves without the keys. The chat servers still see each message in plain text to route it, and they must be able to read the store, so anyone who controls a server, or steals its credentials, reads messages exactly as before. The same holds for every photo in S3. Jane still has to trust every person and system with access to the servers.
GreatEnd-to-end with the Signal protocol
- Keys stay on devices. Each device generates its own identity key pair. It uploads only the public half, with a batch of single-use prekeys, to a key directory.
- Sessions. Before the first message to a device, Jane's phone fetches that device's public
keys with
GET /keys/johnand agrees a shared secret with it. From then on the Signal protocol's ratchet derives a new key for every message, so a key stolen today does not unlock past messages. - One ciphertext per device. In a one-to-one chat, Jane's phone encrypts the message separately for John's phone, John's laptop and her own laptop, and the payload carries all three.
- Groups use sender keys. Each member sends the group a key of their own once, over the one-to-one sessions; after that one ciphertext serves every member, which is why the body can be stored once. When someone leaves, the remaining members switch to new sender keys.
See it in the flythrough16 · End-to-end encrypted
What it changes in the design:
- The server routes ciphertext. Membership, ordering, inboxes, receipts and routing all still work, because they only need the envelope: chat id, seq, sender, devices.
- No server-side features on content. No server search, previews or text spam filters. Search runs on the device.
- Metadata is still visible. The server knows who talks to whom, when, and how big each message is. Say so.
- Trusting the key directory. A compromised server could hand out its own key. Users can compare a safety number derived from both identity keys, and apps warn when a contact's key changes.
Final design
One message, online. Jane's phone sends sendMessage over its WebSocket to chat server A.
Server A takes seq 42 from chat 9's counter, writes the claim for c-881 and the ciphertext in one
transaction, writes an inbox entry for each recipient device, then acks: one tick. It reads
conn:john from the registry and publishes to the channels of servers B and C. Server B pushes the message to John's phone, which
acks; B deletes that inbox entry and sends a delivered receipt back by the same route: two ticks.
John opens the chat, and read { upTo: 42 } turns them blue.
See it in the flythrough09 · Every message
One reconnect. John's phone was offline. The registry had no field for it, so server A called the push service, and APNs showed "1 new message". When the phone reconnects to server B, B writes the registry field, queries the phone's inbox in chat and seq order, and sends 9:44, 9:45 and 12:7, deleting each entry as the phone acks it.
See it in the flythrough19 · End to end
| Component | Role | Scales by |
|---|---|---|
| L4 load balancer | Passes long-lived TCP connections to chat servers | The provider |
| Chat servers | Hold sockets; store, route, push, drain inboxes | 600 servers, ~333,000 sockets each |
| Connection registry | Device to server, for online devices | A Redis cluster |
| Redis pub/sub | One channel per chat server | Shards, ~100,000 publishes / s each |
| Chats and Messages | Members, per-chat counter, ciphertext, 30-day TTL | DynamoDB, keyed by chat and seq |
| Inbox | One small entry per undelivered device and message | DynamoDB, partitioned by device |
| Push service | Wakes offline devices through APNs and FCM | Stateless workers |
| Key directory | Devices' public keys and prekeys | A replicated key-value table |
| Media service, S3, CDN | Upload URLs, encrypted files, downloads | The provider |
What is expected at each level
Junior. A WebSocket per client, a chat server that stores a message and pushes it to online members, an inbox for offline users drained on reconnect, and media through blob storage. With a nudge, you add acknowledgements.
Mid-level. Store before ack, inbox entries deleted by the device's ack, delivered and read receipts, routing between servers through a registry or pub/sub, push notifications for offline devices. You own one deep dive end to end, usually routing between servers with the connection arithmetic.
Senior and above. You start from the durability rule: the inbox is the source of truth, so pub/sub and pushes can be best-effort. You give idempotent at-least-once delivery, a per-chat sequence, per-device inboxes, write-time fan-out justified by the group cap, and an honest account of end-to-end encryption. You name failure modes: a lost zone and its reconnect storm, stale registry entries, a lost publish to a connected device. At the top of this level you also say what a smaller first version could skip (every server subscribing to its users' channels, before a registry), how the design grows to 200M connections, and which trade-offs a team would raise: keepalive frequency against battery and server load, and what end-to-end encryption rules out on the server.
Common mistakes
- Acking before storing. A crash between the two loses a message the sender thinks is sent.
- Polling for new messages. 200M devices polling every few seconds is tens of millions of empty requests a second. Hold a socket and push.
- Sending to every server. Look up where the device is, and publish only there.
- Treating pub/sub as storage. It forgets. The inbox remembers.
- Ordering by timestamps. Phone clocks are wrong and server clocks disagree. Number messages per chat.
- One inbox per user. The laptop never gets the message the phone already acked. Use one per device.
- Saying "end-to-end encrypted" and then reading the message. If the server filters, searches or previews content, it is not end-to-end. Say what the server can see.