Skip to content
SnapFrom

What Actually Happens to a Video Before You See It

· · 5 min read

The mental model most people carry is that a video is a file, the server has it, and watching it means the file arrives over time. That model is about fifteen years out of date, and almost every confusing thing about web video follows from the gap.

What actually reaches you is a few hundred small files, published several times over at different qualities, plus a text document listing where they all are. Nothing in that arrangement is hidden. You can read the text document yourself.

The playlist is just a list

HTTP Live Streaming, the format behind most of what you watch, defines two kinds of playlist. A Media Playlist is one where every URI in it points at a Media Segment. A Master Playlist is one where every URI in it points at another playlist[1].

That second kind is the interesting one. Each entry in a Master Playlist is a Variant Stream: the same content “encoded at a particular bit rate, in a particular format, and at a particular resolution”[1]. Six entries in a master playlist means the video was encoded six times and all six exist on disk right now, waiting to see which one you ask for.

At the start of a session the player downloads that playlist, which “contains the metadata for the various sub-streams that are provided”[2]. Everything after that is the player fetching numbered segments over ordinary HTTP.

The segments are short on purpose

A playlist declares a target duration, and the specification requires that every segment in it, rounded to the nearest second, be no longer than that target[1]. In practice that puts most segments in the two-to-ten-second range.

Short segments are what make the whole thing work. The player is never committed to more than one segment at the current quality, so it can change its mind roughly every few seconds. Long segments would mean better compression and fewer requests, and a player that is stuck with a bad decision for half a minute.

For live streams there is a second constraint that explains a lot of stalling behaviour: a client “SHOULD NOT choose a segment that starts less than three target durations from the end” of the playlist[1]. That gap is deliberate. It is the buffer that absorbs a slow response, and it is also most of the reason a live stream is behind the event it is showing.

Nobody specified how it chooses

Here is the part that surprises people. The specification defines the playlist format, the segment constraints and the alignment rules, and then says outright that the algorithms a client uses to switch between Variant Streams “are beyond the scope of this document”[1].

There is no standard answer to the central question. Every player vendor decides for itself, and they disagree.

What they are all doing is some version of the same guess: “the data transfer rate is monitored and if it looks like it’s not keeping up, we drop down to a lower bandwidth (and consequently lower quality) stream”[2]. Monitoring what already happened and betting the next few seconds resemble the last few is the entire mechanism. It is not a measurement of your connection. It is an extrapolation from recent downloads.

This is why quality drops at moments that feel wrong. The estimate is backward-looking, so a brief stall, a competing download, or a shift between cell towers all read the same way: the last segment was slow, so the next one should be smaller. By the time the connection recovers, the player has already committed to a lower rung and has to climb back up one segment at a time.

Why the switch is invisible

Switching mid-playback only works because the variants are constrained to line up. The specification requires that matching content in Variant Streams have matching timestamps and matching discontinuity sequence numbers, and that each Variant Stream present the same content[1].

Note what that does and does not say. The variants are not identical files at different sizes. They are separately encoded, so a given frame genuinely differs between them. What must match is the timeline, so that segment 47 of the 1080p ladder and segment 47 of the 480p ladder cover the same seconds of the same content. That is what lets the player splice them together without a seek.

The browser is doing this in JavaScript

None of this is built into the <video> element. Media Source Extensions is what makes it possible: MSE “allows us to replace the usual single progressive src URI fed to media elements with a reference to a MediaSource object”, holding references to multiple SourceBuffer objects that represent the different chunks of media making up the stream[3]. MDN is explicit that this “lays the groundwork for adaptive bitrate streaming clients (such as those using DASH or HLS) to be built on its extensible API”[3].

So on most sites, the thing deciding what quality you get is a JavaScript library shipped with the page. It fetches segments, it appends bytes into a buffer, and it holds “finer-grained control over how much and how often content is fetched”[3]. The video element is downstream of all of it.

What this costs you

Three things follow, and they are worth naming because none of them is a bug being fixed.

The quality you see is a prediction, not a setting. Unless you pin it manually, you are watching whatever a heuristic decided from your last few seconds of throughput. Two people on the same connection, on the same video, at the same moment, can legitimately be watching different encodes.

Live is structurally late. The three-target-duration rule is a floor, not a tuning knob. A stream with six-second segments cannot be less than roughly eighteen seconds behind and still follow the specification.

The player is a program the site chose. Its switching logic is unspecified, so its behaviour is a vendor decision that you cannot inspect from the outside and that changes when the site updates its library.

Nothing here is a trick being played on you. It is what it takes to deliver video over a protocol that was designed for documents, to a connection that nobody can measure in advance.

Sources

  1. RFC 8216 - HTTP Live Streaming - IETF, accessed

    Supports: A Media Playlist is one where all URI lines identify Media Segments, while a Master Playlist is one where all URI lines identify Media Playlists, each describing a Variant Stream - a version of the same content 'encoded at a particular bit rate, in a particular format, and at a particular resolution' (section 2). Section 4.3.3.1 requires that the EXTINF duration of every segment, rounded to the nearest integer, MUST be less than or equal to the playlist's target duration. Section 6.3.3 says a client SHOULD NOT choose a segment that starts less than three target durations from the end of a live playlist. Section 6.3.1 states that the algorithms a client uses to switch between Variant Streams are beyond the scope of the document. Section 6.2.4 requires that matching content in Variant Streams have matching timestamps and matching discontinuity sequence numbers, and that each Variant Stream present the same content.

  2. Live streaming web audio and video - MDN Web Docs, Mozilla, accessed

    Supports: Livestreaming formats 'generally allow adaptive streaming by breaking streams into a series of small segments and making those segments available at different qualities and bit rates', and the adaptation itself works because 'the data transfer rate is monitored and if it looks like it's not keeping up, we drop down to a lower bandwidth (and consequently lower quality) stream'. At the start of a session an extended M3U playlist is downloaded that contains the metadata for the various sub-streams provided.

  3. Media Source Extensions API - MDN Web Docs, Mozilla, accessed

    Supports: MSE 'allows us to replace the usual single progressive src URI fed to media elements with a reference to a MediaSource object', a container holding references to multiple SourceBuffer objects representing the different chunks of media that make up the stream, and it 'lays the groundwork for adaptive bitrate streaming clients (such as those using DASH or HLS) to be built on its extensible API'. It also gives 'finer-grained control over how much and how often content is fetched'.