docs/ORIN-RUNTIME.md

---
type: default
---
# Orin runtime

[ARCHITECTURE.md](ARCHITECTURE.md) describes services and the bus without saying
how they coexist on one 8 GB board. This document covers that: what runs, in what
order, under what budget, and what happens when it goes wrong.

Cross-cutting by nature — it spans `svc-core`, `svc-nav`, `svc-gw` and the host
repo, so it lives in the hub rather than in any one of them.

## Layers

```
  clients        app-flutter, app-esp32          off the Orin
  ─────────────────────────────────────────────────────────
  serving        svc-gw, svc-nav
  processing     decoder, recorder
  acquisition    can-bridge, obd-poller, gnss
  bus            mosquitto
  ─────────────────────────────────────────────────────────
  host           kernel modules, gpsd, PipeWire, network     orin-nano-env-setup
```

A layer may depend on the one below it and on the bus. Nothing depends sideways:
no service waits on another service's process, only on the broker.

## The binding constraint: 8 GB, shared

The Orin Nano has 8 GB of LPDDR5 **shared between CPU and GPU**. There is no
separate VRAM to fall back on — a model loaded onto the GPU takes memory away
from everything else on the board. Roughly 6–7 GB is left once JetPack and the OS
have taken their share.

| Consumer | Resident | Note |
|---|---|---|
| OS + JetPack | ~1–1.5 GB | before anything of ours starts |
| mosquitto, acquisition, `svc-gw` | tens of MB | not a factor |
| Valhalla server | small; tiles are **mmap'd page cache** | evictable — see below |
| Whisper | 273 MB (tiny) → 3.9 GB (large) | `small` is 244 M params; `large` does not fit |
| GPT-SoVITS | upstream guidance assumes an 8 GB VRAM card | the largest unknown; **measure before designing around it** |
| vision (Phase 5) | — | competes directly with voice for the same pool |

Four rules follow.

**The cloud LLM is load-bearing.** [OVERVIEW.md](../OVERVIEW.md) puts reasoning in
the cloud. That decision is what makes a local voice stack fit at all — moving the
LLM on-device collapses this budget entirely. It is not a free choice to revisit.

**Size Whisper by vocabulary, not by benchmark.** In-car commands are a narrow,
predictable set. `large` buys accuracy this application cannot spend, at a cost
this board cannot pay.

**Voice and vision cannot both be resident.** Phase 5 has to plan for eviction,
not addition. Worth knowing now, while vision is still deferred.

**Valhalla's memory-mapped tiles are not a leak.** They appear as page cache and
are evictable. Under pressure the kernel drops tile pages and re-faults them on
next use — that costs routing latency, not an OOM. This is the graceful failure
in the system and should be left alone.

## Startup tiers

Ordered by what is lost if a tier is late.

| Tier | Units | Why here |
|---|---|---|
| 0 | kernel modules, gpsd, network | Host layer; nothing of ours can precede it |
| 1 | `mosquitto`, `can-bridge`, `recorder` | **Data is lost if late.** Must reach ready fastest |
| 2 | `decoder`, `gnss`, `obd-poller` | Derive from tier 1; a late start loses nothing |
| 3 | `svc-gw`, `svc-nav` | Serving; a display can wait |
| 4 | voice — ASR, TTS | Interactive; seconds late is harmless |

**Capture first** is the whole principle. A CAN frame missed at tier 1 is gone
forever; a map that loads five seconds into the drive costs nothing. Ordering the
boot the intuitive way — get the screen up quickly — trades irreplaceable data for
a cosmetic gain.

## Failure isolation

- `Restart=always` on every unit. A service dying mid-drive is a normal event.
- Units order themselves `After=` the broker and nothing else. No service may wait
  on another service, or a restart cascades.
- The broker is the single point of failure, and that is accepted: retained topics
  make reconnection cheap, so a client that comes back has state on its first
  message rather than waiting for the next publish.
- No service holds another's state. Anything worth surviving a restart is either
  retained on the bus or on disk.

## Ignition off

Shutdown is the reverse of startup, so tier 1 dies last.

The only thing that *must* complete is the recorder's flush. It therefore owns its
own flush and shutdown handling rather than trusting the OS to get there in time —
a truncated session is the one failure that cannot be recovered by restarting
anything.

## Storage

Self-built binaries live under `/home/mikkeli/infra/` per the host repo's
guidelines. Data does not.

| Data | Size | Growth |
|---|---|---|
| Map artifacts | GB | Replaced wholesale, quarterly at most |
| Voice models | GB | Static |
| Recorded sessions | GB per drive | **Unbounded** |

Recordings are the problem. Without a retention policy the disk fills mid-drive
and capture stops — silently, because a full disk is not a crash. A retention
policy is a tier-1 concern, not housekeeping.

## Audio

The Orin owns the car's single aux input, via a USB DAC.

```
  GPT-SoVITS (TTS) ──┐
                     ├──▶ PipeWire mixer ──▶ USB DAC ──▶ aux in
  A2DP sink ─────────┘         │
  (media from the tablet)      └── ducking: media drops under TTS

  mic ──▶ Whisper (ASR)
```

Three reasons the aux cable belongs here rather than on the tablet:

- **The aux cable should belong to the fixed unit.** The display is meant to be
  removable; anything holding a permanent cable to the car is not removable.
- **Ducking becomes a local policy** on one box instead of coordination between
  two. Without it the assistant talks over the music.
- **The direction works.** Linux is a capable A2DP *sink*, so the tablet can send
  media here. The reverse — this box as an A2DP source into an Android sink — is
  not viable, since stock Android builds generally do not enable A2DP sink.

Unplug the tablet and the car keeps audio and voice. The Orin depends on the
display unit for nothing.