Google Drive / Dropbox

What is being asked

Design a file hosting and sync service. A file changed on a laptop appears on the phone; files can be shared; large files upload reliably over poor connections.

Requirements

Functional

• Upload, download, delete; folders; sharing by link or with users.
• Sync changes to all of a user's devices; keep version history.
• Work offline and reconcile later.

Non-functional

• Files up to many gigabytes; resumable transfers.
• Do not lose data. Durability matters more than anything else.
• Use bandwidth and storage efficiently.

Design

1. The client splits a file into blocks of a few megabytes and hashes each block.
2. It asks the metadata service which hashes are already stored and uploads only the missing blocks, directly to object storage.
3. The metadata service then commits a new file version: an ordered list of block hashes. This commit is a transaction.
4. Other devices hold a long-lived connection to the notification service. On a change they fetch the new metadata and download only the blocks they lack.

Key decisions and trade-offs

• Separate metadata from content. Metadata needs transactions and fast queries; content needs cheap, durable bulk storage. They scale differently.
• Block-level de-duplication saves storage and makes a one-byte edit to a large file a small upload. Content-defined chunking keeps blocks stable when bytes are inserted.
• Conflicts. If two devices edit the same version offline, keep both and name one a "conflicted copy". Silent merging risks losing work.
• Consistency. Metadata must be strongly consistent, or devices see a file list that disagrees with the stored blocks.

Follow-up questions

• How do you delete blocks that no file refers to any more?
• How would you encrypt files so the service cannot read them, and what does that cost in de-duplication?
• How does a device that was offline for a month catch up?
• How do you share a folder of a million files efficiently?