brfc-stream-pirate/README.md

# brfc-stream-pirate

Personal time-shifting archive for Bristol Rovers video pass streams.
Kickoffs land at unfriendly hours in JST; this records them unattended.

Stream is **plain HLS, no DRM** (StreamAMG behind Akamai). Nothing here breaks
a protection measure -- it replays the same authenticated request the player
makes, using your own paid session.

## Pieces

| File | Job |
|---|---|
| `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR in the page, captures the manifest URL verbatim, POSTs it to the catcher |
| `catcher.py` | Listens on 127.0.0.1:8787, hands the URL to yt-dlp, records to `recordings/` |
| `probe.py` | Byte-exact manifest fetch + concurrency test |

## The token

Manifest URLs carry an Akamai path token:

    hdntl=exp=<unix>~acl=/*~id=<n>~data=hdntl~hmac=<sig>

- `exp` -- ~24h validity
- `acl=/*` -- wildcard over the whole host, not just your match
- `hmac` -- signature over the literal path

It is a bearer credential. Anyone holding a live URL can pull the stream until
`exp`. Keep it out of pastebins and issue trackers. `stream-url.txt` and
`cookies.txt` are gitignored.

**Do not re-encode the path.** curl normalises `%2f` to `%2F`, which
invalidates the HMAC. `probe.py` sends the request target verbatim.

## Use Firefox for watching

yt-dlp lifts session cookies from the browser. Chrome 127+ on Windows uses
app-bound cookie encryption and extraction fails. Firefox works with no fuss.

    set BRFC_BROWSER=firefox     # default

## Workflow

    pip install yt-dlp                    # done
    python catcher.py                     # leave running

Install the userscript in Tampermonkey, open the match page, press play. The
sniffer catches the manifest, the catcher starts yt-dlp. Recording is MPEG-TS
(`--hls-use-mpegts`) so a killed process still leaves a playable file.

For a 3am kickoff: Task Scheduler opens the match page a few minutes before,
the browser autoplays, the sniffer fires, the catcher records. Nobody is awake.

## Concurrency check

An Akamai edge token is a stateless HMAC -- the edge validates signature and
expiry and serves; it holds no session table, so it cannot count viewers. Any
concurrent-stream limit lives in the entitlement API that *issues* tokens,
upstream of the CDN. Confirm rather than assume, while the browser is playing:

    python probe.py --url "<manifest from the sniffer>" --parallel 3

3/3 alongside the playing browser means four concurrent readers of one token
and the edge does not care.

---

## Findings (2026-08-29, live 2300 JST match)

Verified against a live stream. Recorded 1080p successfully end to end.

### Two token types, not one

The player uses a **two-stage token chain**:

| Token | Where | TTL | Purpose |
|---|---|---|---|
| `hdnts=` | query string, `acl=/*` unencoded | **~25 seconds** | fetch the master playlist |
| `hdntl=` | URL path, `acl=%2f*` encoded | **~24 hours** | media playlists + segments |

The master response rewrites each variant URL with a 24h `hdntl` token. So the
`hdntl` token is the durable one worth handing to a downloader. Chasing a
`hdnts` URL by hand is pointless -- it dies before you can paste it.

### Confirmed by experiment

- **Concurrency is not limited.** Two independent ffmpeg downloads ran at full
  bitrate *while Firefox played the same match*. Three concurrent readers of
  one token, no complaint from the edge. Akamai token auth is a stateless HMAC
  check; it has no viewer counter.
- **No CDN cookies exist.** All 13 relevant cookies are on `bristolrovers.co.uk`;
  nothing is scoped to `akamaized.net`. The token in the URL is the whole auth.
- **Stream is 1920x1080 H.264 + AAC ~5.2 Mbit/s** -> roughly 4.7 GB per match.
- **Output is MPEG-TS** even though yt-dlp names it `.mp4`. That is what
  `--hls-use-mpegts` buys: a killed process still leaves a playable file.
- Player is **Bitmovin**, embedded on the club site. `0_xm5vrrw6` is a Kaltura
  entry id and identifies the match -- the catcher dedupes on it.

### Gotcha: which URL form a Python client can fetch

Earlier notes here blamed a TLS handshake difference. That was wrong. The 403
splits by **token form**, not by client:

| URL form | Python | ffmpeg |
|---|---|---|
| `index.m3u8?hdnts=...` (master, query token) | 200 | 200 |
| `hdntl=.../index_N.m3u8` (media, path token)  | 403 | 200 |

Consequences:

- Mint a **master** URL and hand that to the downloader. `record_live.py` does
  this: it calls the entry API for a fresh master immediately before recording.
- Force `--downloader ffmpeg`. yt-dlp's native HLS downloader fetches variant
  playlists with Python and dies on the path-token URLs. This is required, not
  a preference.
- `probe.py` is diagnostic scaffolding and inherits the path-token 403 -- do
  not use it to conclude a token is dead.

### Firefox, not Chrome

yt-dlp reads cookies from a running Firefox. Chrome/Edge lock their cookie DB
while running (yt-dlp issue 7271), and an unattended job cannot close the
browser to read them.

### The stream never ends

After full time the feed switches to a static league slate and keeps serving
indefinitely at full CBR bitrate (~4.7 Mbit/s to encode a still image). There
is no `#EXT-X-ENDLIST`, so ffmpeg never finishes, and there is no stall to
detect either -- from the downloader's point of view it is a healthy live
stream forever.

**Keep the duration cap.** `catcher.py` and `record_live.py` pass
`ffmpeg_o:-t $DURATION_MIN*60`. Without it, a feed that behaves like the video
one above fills the disk with a logo.

Note the sample size. This is **one video observation**, and it is contradicted
by **one audio observation**: on 2026-09-02 an audio stream ended cleanly by
itself (`rc=0` at 165 min, against a 180 min cap that never fired). So "the
stream never ends" is a property of that video feed, not an established rule.
Treat the cap as a safety net rather than as the only stop condition.

Observed: video, 116 min recorded for a 23:00 kickoff, still running at 01:04
and stopped manually. Audio, 165 min, self-terminated. Budget ~3.5-4 GB per copy per match at 40 MB/min.

---

## Status

Unattended recording works. A scheduled task started `record_live.py` at 03:15
for an 03:45 kickoff, found the stream in 16 seconds, and recorded 165 min of
audio to a playable MPEG-TS with nobody awake.

Sample sizes are one. One video capture, one audio capture. Where the two
disagree (stream termination, whether an entry id persists between matches),
neither is established -- do not build on either without more matches.

### Verified

- Token chain end to end in plain Python: session cookie -> JWT -> entry API
  -> master playlist. No browser, no fingerprint forging.
- Recording via ffmpeg, video (1080p) and audio (~122 kbps AAC).
- Scheduled unattended run, including that Task Scheduler launches it correctly.
- Concurrency: several readers of one token, alongside a playing browser.

### Not verified

- **Login.** Everything above rides an `SSESS` cookie lifted from a desktop
  Firefox profile. Obtaining a session *from credentials* has never been run.
  `/user/login` is a standard Drupal form (`name`, `pass`, `form_build_id`,
  `form_id`, `captcha_token`) with no reCAPTCHA script and no 2FA mention, but
  `captcha_token` is an unknown and testing it risks a lockout that would take
  out the working session too.

### Decision: do not automate login

The cookie lasts ~87 days and matches are weekly, so automating login saves
about four manual actions a year -- while being the riskiest step and the only
one needing a stored password. Refresh the cookie by logging in manually on the
desktop each quarter and provisioning it to the recording host instead. That
makes cookie *provisioning* the open problem, not the captcha.

Use Firefox as the session holder. Two independent reasons: its cookie DB stays
readable while running (Chrome/Edge lock theirs), and it holds the login
durably where Chrome has been observed dropping it within a day.