YouTube Video Ingestion

What is being asked

Design the upload and processing side of a video platform, from the creator pressing "upload" to the video being playable worldwide.

Requirements

Functional

• Upload large files reliably; process into several resolutions and formats.
• Generate thumbnails; run checks; publish.

Non-functional

• Files of many gigabytes over unreliable connections.
• Processing is CPU-heavy; a short video should be available in minutes.
• A failure halfway must not restart the whole job.

Design

1. The client uploads in chunks directly to object storage with a resumable protocol; a dropped connection continues from the last completed chunk.
2. Completion places a job on a queue. The pipeline is a graph of tasks, each idempotent and retried independently.
3. The video is split into segments of a few seconds at key frames. Many workers transcode segments in parallel into each target resolution and codec.
4. Outputs are stored as small segment files with a manifest listing the available qualities. Players switch quality between segments as bandwidth changes (adaptive bitrate streaming).
5. Thumbnails, content checks and metadata extraction run alongside. The video is published when the minimum set of renditions is ready.

Key decisions and trade-offs

• Split then transcode in parallel turns an hour-long video from an hour of work into minutes on many machines.
• Publish low resolutions first. Viewers can watch at 360p while 4K is still encoding.
• Queue-driven workers on cheap, interruptible machines. Tasks are small and repeatable, so losing a worker costs little.
• Storage cost. Many renditions multiply storage; rarely watched videos can keep fewer and encode others on demand.

Follow-up questions

• How would you prioritise a live news upload over a backlog?
• How do you avoid re-encoding a file that was uploaded twice?
• What is different for live streaming?
• How do you choose the set of resolutions and bitrates?