Design YouTube Upload and Playback

Updated · By

Upload in chunks to a resumable session so a dropped connection resends only the missing parts. Split the video into segments, transcode them in parallel into a ladder of resolutions and bitrates, and publish as soon as the first rendition is ready. Players fetch short segments from a CDN and switch bitrate as bandwidth changes.

hardPattern: CachingFrom Devana's Google question set

Problem

Design the path from a creator pressing upload to a viewer on another continent pressing play.

Cover ingesting a very large file over an unreliable connection, transcoding it into the formats every device needs, storage, and global delivery.

POST /uploads                           {size, filename}  -> {upload_id, chunk_size}
PUT  /uploads/{upload_id}/chunks/{n}    <bytes, checksum>  -> {ok}
GET  /uploads/{upload_id}               -> {missing_chunks}
POST /uploads/{upload_id}/complete      -> {video_id}
GET  /videos/{video_id}/manifest        -> renditions and their segment URLs

This is one of Devana's own questions, written in the style of Google's interviews. It is not a question leaked from Google, and Devana is not affiliated with Google.

Requirements

  • Uploads of up to 100 GB, over connections that drop.
  • A video should be watchable within minutes of upload, not hours.
  • Viewers on connections from 200 kbps to 1 Gbps.
  • Playback starts in under 2 seconds at the 95th percentile.
  • Worth asking first: is live streaming in scope? Assume not: uploaded videos only.

Scenarios

Scenario 1

A creator's connection drops 60 GB into a 100 GB upload. Expected: The upload resumes from where it stopped; the 60 GB already sent is kept.

The client asks which chunks the server has and sends only the rest. Each chunk counts as received only once its checksum matches, so a chunk cut off mid-transfer is simply sent again.

Scenario 2

A viewer on a 300 kbps mobile connection presses play. Expected: Playback starts quickly at a low rendition and steps up if the connection improves.

The player picks a rendition from the manifest by its measured bandwidth, fetches a few seconds of video at a time from a nearby CDN edge, and can switch rendition at every segment boundary.

Hints

Open one at a time, and try the problem again after each.

Hint 1

A 100 GB upload will fail partway at least once. What lets it continue without starting again?

Hint 2

Transcoding a long video in one piece takes hours. What can you split it into, and run at the same time?

Hint 3

Viewers range from 200 kbps to 1 Gbps. What does the player need so that both get smooth playback?

Hint 4

Which bytes do most views touch first, and where should those bytes live?

A worked answer

Requirements and estimates

Assume 10^6 uploads a day averaging 1 GB: 10^15 bytes, a petabyte a day of source video, before the renditions add more. Playback dominates: assume 10^9 views a day of 5 minutes at an average of 2 Mbps. That is 75 MB a view, 7.5 × 10^16 bytes a day, on the order of 7 terabits a second on average.

No set of origin servers serves that. Delivery has to come from caches near viewers, and the origin should only see misses. Every later choice follows from that one number.

Resumable upload

The client opens an upload session and sends the file in fixed-size chunks; with 64 MB chunks, 100 GB is about 1,600 chunks. Each chunk carries a checksum, and the server records it as received only when the checksum matches. After a drop, the client asks which chunks are missing and sends just those, several at a time.

The bytes go straight to object storage through signed URLs, so no application server carries 100 GB. The upload service only keeps the session: which chunks have arrived. When every chunk is in, it starts processing.

Transcoding in parallel

Split the source at keyframes into segments a few seconds long, and put one job per segment and rendition on a queue. A one-hour video in 4-second segments is 900 segments; with six renditions from 240p to 4K, that is 5,400 jobs that can run at the same time, so the wall-clock time is minutes rather than hours.

Do the lowest renditions first and publish as soon as one rendition is complete. The manifest, the list of renditions and their segments in a standard format such as HLS or DASH, gains the higher renditions as they finish.

Storage

Keep the source in object storage on a cheaper tier, so the video can be transcoded again when a new format arrives. Renditions and their segments live in object storage too. A relational table holds each video's metadata and which renditions are ready, which is what the playback API reads to build the manifest.

Global delivery

Viewers fetch segments from CDN edges near them. The player measures its throughput and picks a rendition for each segment, starting conservatively so the first frame arrives fast; the first seconds of each rendition are the hottest bytes, so cache those widely to meet the 2-second start.

Place videos by predicted demand: a creator's subscribers by region, early views, what is trending locally. Predicted videos are pushed to edges ahead of time; the long tail stays in regional caches and the origin and is fetched on a miss.

The whole design at a glance

The design for Design YouTube Upload and PlaybackCreatorClientViewerClient5Upload serviceService1Source uploadsObject storePlayback APIServiceCDN4Transcode jobsMessage queueVideo metadataSQL database3Transcode worke…Worker2RenditionsObject storechunk statuschunks, signed URLssplit into segmentssource segmentssegments per renditionrendition readyHTTP · segmentson a missmanifest
The design above in one picture; dashed arrows are asynchronous. The numbered parts are the choices that matter most, each explained above: 1. Upload service; 2. Transcode workers; 3. Video metadata; 4. CDN; 5. Viewer.

How to say it in the interview

Split the answer into three pipelines, upload, processing and playback, and name the constraint each one answers: resumable chunks for the dropped connections, parallel segment transcoding for minutes rather than hours, and a CDN with adaptive bitrate for the 2-second start across 200 kbps to 1 Gbps. Put a number on the egress early; it is the reason the CDN carries the design.

The core in code

The upload session's bookkeeping, which is what lets a 100 GB upload survive a dropped connection: a chunk counts only when its length and checksum match, the client asks what is missing after a reconnect, and the upload completes only when every chunk has arrived.

Python

import hashlib

class UploadSession:
    """A resumable upload: the file in fixed-size chunks, each one
    recorded only after its length and checksum match."""

    def __init__(self, size, chunk_size):
        self.size = size
        self.chunk_size = chunk_size
        self.count = -(-size // chunk_size)      # ceiling division
        self.received = {}                       # chunk number -> checksum

    def expected_length(self, n):
        if n == self.count - 1:
            return self.size - n * self.chunk_size   # the last chunk may be short
        return self.chunk_size

    def put(self, n, data, checksum):
        if not 0 <= n < self.count or len(data) != self.expected_length(n):
            return False
        if hashlib.sha256(data).hexdigest() != checksum:
            return False                         # damaged in transit: send it again
        self.received[n] = checksum              # a repeated chunk is harmless
        return True

    def missing(self):
        return [n for n in range(self.count) if n not in self.received]

    def complete(self):
        return not self.missing()

JavaScript (Node.js)

const crypto = require("node:crypto");

class UploadSession {
  // A resumable upload: the file in fixed-size chunks, each one
  // recorded only after its length and checksum match.
  constructor(size, chunkSize) {
    this.size = size;
    this.chunkSize = chunkSize;
    this.count = Math.ceil(size / chunkSize);
    this.received = new Map(); // chunk number -> checksum
  }

  expectedLength(n) {
    // The last chunk may be short.
    return n === this.count - 1 ? this.size - n * this.chunkSize : this.chunkSize;
  }

  put(n, data, checksum) {
    if (!Number.isInteger(n) || n < 0 || n >= this.count) return false;
    if (data.length !== this.expectedLength(n)) return false;
    const actual = crypto.createHash("sha256").update(data).digest("hex");
    if (actual !== checksum) return false; // damaged in transit: send it again
    this.received.set(n, checksum); // a repeated chunk is harmless
    return true;
  }

  missing() {
    const out = [];
    for (let n = 0; n < this.count; n++) if (!this.received.has(n)) out.push(n);
    return out;
  }

  complete() {
    return this.missing().length === 0;
  }
}

Both versions are tested against the same cases, including every example on this page.

  • A chunk is recorded only after the length and checksum check, so a chunk cut off by a dropped connection never counts, and the client sends it again.
  • Putting the same chunk twice is harmless, which matters because a client that lost the reply cannot know whether its last chunk arrived.
  • In production the session lives in a database, the bytes go to object storage, and complete triggers the processing pipeline.

Follow-up questions

How do you make a 100 GB upload resumable after a dropped connection?

Split the file into fixed-size chunks under an upload session. The server records each chunk once its checksum matches; after a drop, the client asks which chunks are missing and sends only those, several at a time. Chunks go straight to object storage through signed URLs, so no application server carries the bytes.

Do you transcode the whole video before publishing, or can you publish sooner?

Publish sooner. Transcode segments in parallel, lowest renditions first, and mark the video watchable once one complete rendition exists; the higher ones join the manifest as they finish. Viewers in the first minutes may only get the lower qualities.

How do you decide which videos to push to which CDN edges?

By predicted demand: the creator's subscribers by region, early view velocity and what is trending locally. Predicted videos are pushed to edges in those regions ahead of time, and the long tail is fetched from regional caches or the origin on a miss.

A video unexpectedly goes viral in one country. What happens?

Edges there miss at first, so regional caches and request coalescing at the edge make sure each segment is fetched once rather than by every viewer at the same moment. As views climb, the demand signal places the video on more edges in that country, lowest renditions first.

Common mistakes

  • Uploading the file in one request, so any drop restarts all 100 GB.
  • Transcoding the whole video as one job and publishing only when every rendition is done.
  • One bitrate for everyone, which stalls slow connections and wastes fast ones.
  • Serving video from origin servers, with no plan for the egress the numbers show.
  • Sending upload bytes through application servers instead of straight to storage.

Practice it in a mock interview

Reading a solution and explaining one under questioning are different skills. The button below starts this problem as a voice mock interview with Josh, Devana's AI interviewer, with a whiteboard he can read.

Practice this problem in a mock interview

Practice this problem out loud

Josh, Devana's AI interviewer, runs this exact problem as a voice mock interview, asks follow-ups like the ones above, and scores you out of 100. The Free plan has 3 interviews a month, no card needed.

Practice this problem in a mock interview