svc-core/docs/impl/reports/BUS-BENCH.md

---
type: report
date: 2026-10-12
---
# Raw CAN over MQTT: bus bench

Can mosquitto carry `yukart/can/raw` at real CAN rates (project-yukart decision 4)?
Measured on the car Orin (mosquitto 2.0.11, localhost) with `bus-bench`: `bridge`
reads a `vcan0` and publishes frames, `sink` subscribes, checks sequence numbers
and measures latency from the kernel's reception timestamp to the subscriber.
The car's usual services (recorder, orin-status, svc-gw, svc-vision) kept running.

```bash
scripts/bus-bench.sh ./bus-bench z27ag-20261009-163736-ign.log 20     # on the Orin
scripts/bus-mix.sh   ./bus-bench z27ag-20261009-163736-ign.log 20     # after bus-bench.sh
```

Loads:

- **car replay**: the in-car capture (`z27ag-20261009-163736-ign.log`, engine at
  idle) at its own timing, about 890 frames/s. This is the real bus.
- **full bus**: `cangen -L 8` at about 4,400 frames/s. 500 kbit/s carries roughly
  3,700–4,500 8-byte frames/s depending on bit stuffing, so this is a saturated bus.
- **flood**: `cangen -g 0`, as fast as vcan goes, to find where it breaks.

Two encodings: one MQTT message per frame, or frames batched over a 10 ms window.
Both use the same provisional binary format (24 bytes a frame, 12-byte header;
see `src/bus-bench/main.rs`). QoS 0, not retained.

## Results (20 s runs; full bus 15 s)

With `TCP_NODELAY` on both the clients and the broker (see below):

| Load | Encoding | Frames/s delivered | Lost | Latency p50 / p99 / max (ms) | CPU: broker · bridge · sink (% of a core) |
|---|---|---|---|---|---|
| car replay | per frame | 893 | 0 | 0.31 / 0.92 / 1.3 | 4.7 · 3.5 · 3.8 |
| car replay | 10 ms batch | 893 | 0 | 8.7 / 15.3 / 17.6 | 0.9 · 1.2 · 0.6 |
| full bus | per frame | 4,440 | 0 | 0.16 / 0.30 / 3.3 | 27.5 · 21.7 · 17.0 |
| full bus | 10 ms batch | 4,562 | 0 | 5.3 / 10.4 / 10.8 | 1.1 · 3.5 · 0.8 |
| flood | per frame | 45,700 | 84 % | ~1.8 s, queues full | 85 · 86 · 81 |
| flood | 10 ms batch | 268,800 | 0 | 5.3 / 10.2 / 11.1 | 1.3 · 63 · 0.9 |

- **Real rates hold, with margin.** Per frame breaks at about 45,000 messages/s:
  ten times a saturated bus and fifty times the real one.
- **Batching is the clear choice.** At a full bus it costs the broker about 1 % of a
  core instead of 28 %, at a fixed ~100 messages/s whatever the bus load. Frames are
  never lost: the batched flood (60 times a full bus) still lost none, and the
  bridge's own CAN reads became the limit. The price is up to one window of
  latency (the 10 ms here). That is no problem for a dashboard, a recorder or a
  decoder.
- Per frame stays the option for a consumer that needs sub-millisecond latency. Nothing
  planned does.

## Nagle: the 40 ms tail

The first runs had a p99.9 near 40 ms at every load: Nagle's algorithm holding small
writes back until the previous one is acknowledged, against delayed ACKs. Both
sides do it by default: rumqttc (`NetworkOptions::set_tcp_nodelay`, off by default)
and mosquitto (`set_tcp_nodelay`, off by default).

| Full bus (~3,000 frames/s) | per frame p99 / p99.9 / max | 10 ms batch p99 / max |
|---|---|---|
| Nagle on both sides | 23.5 / 41.7 / 45 | 65.6 / 80.9 |
| client no-delay, broker Nagle | 0.61 / 38.0 / 47 | 43.3 / 55.3 |
| no-delay on both | 0.33 / 0.67 / 2.8 | 10.5 / 10.8 |

The car's broker now has `set_tcp_nodelay true` (orin-nano-env-setup `MOSQUITTO.md`).
Clients that publish or consume raw CAN set `set_tcp_nodelay(true)` too. At once a
second (orin-status) it does not matter.

## The bus as the car will use it

`scripts/bus-mix.sh`: the raw stream (10 ms batches) to four subscribers (the decoder
plus three: svc-gw, telemetry-gw, a debug tool), and a stand-in `decoder` publishing
envelopes on one retained topic per signal, to five subscribers that each take every
signal (svc-gw, recorder, svc-nav, svc-vision, telemetry-gw). Loads keep the car's
27 CAN IDs and their rates (2–100 Hz): the capture as recorded, and played 5× faster
for a full bus. Signals per frame and a per-signal rate cap (latest value wins)
stand in for the decoder design nobody has made yet. 20 s runs.

| Load | Signals/frame | Cap | Envelopes/s | Deliveries/s (×5) | Broker CPU | Envelope latency p50 / p99 (ms) |
|---|---|---|---|---|---|---|
| car | 3 | 50 Hz | 1,600 | 8,000 | 22 % | 13 / 21 |
| car | 3 | none | 2,550 | 12,800 | 35 % | 13 / 20 |
| car | 8 | 20 Hz | 2,550 | 12,800 | 32 % | 14 / 27 |
| car | 8 | 50 Hz | 4,300 | 21,600 | 47 % | 16 / 32 |
| car | 8 | none | 6,800 | 34,000 | 51 % | 14 / 47 |
| full bus | 3 | 50 Hz | 3,100 | 15,500 | 39 % | 12 / 18 |
| full bus | 8 | 20 Hz | 3,500 | 17,300 | 44 % | 22 / 41 |
| full bus | 8 | 50 Hz | 8,100 | 40,600 | 57 % | 15 / **391** |
| full bus | 3 | none | 12,800 | 63,800 | 74 % | **636 / 1,472** |
| full bus | 8 | none | 16,300 offered | — | 75 % | **5,380 / 5,994**; decoder queue overflowing |

- **The raw stream never suffered.** All four raw subscribers got every frame in every
  run, p99 ≤ 16 ms, including the runs where the envelopes collapsed.
- **Envelopes are the load.** The broker's cost follows deliveries (messages ×
  subscribers). Up to ~22,000 deliveries/s it stays under half a core with p99 near
  30 ms; around 40,000 the tail goes to hundreds of ms; past ~60,000 queues grow by
  seconds. Mosquitto is single-threaded, so this is a ceiling, not a share.
- **Each subscriber pays too.** Parsing JSON envelopes cost each sink 14–38 % of a
  core at these rates; five of them took two cores. A real subscriber that takes
  everything (svc-gw) pays the same.
- Envelope loss is not measured exactly (no sequence numbers in the envelope);
  the decoder's own `refused` counts what it could not even queue.

What keeps it in the comfortable range, any of which is enough at the car's rate:

1. **Cap each signal's rate in the decoder**, latest value wins. 20–50 Hz is still
   10–50× OBD-II polling, and the raw stream keeps the full rate for anything that
   needs it.
2. **Subscribe narrowly.** Only a consumer that shows or stores everything (svc-gw)
   needs `yukart/vehicle/+`; svc-nav and svc-vision need a handful of topics.
3. **Publish slow signals on change**, with a heartbeat; temperatures and states do
   not need 50 Hz.
4. **The recorder need not record decoded signals at full rate.** It records raw
   frames, and decoding them again offline gives the same signals.

## Not tested

- A real CAN adapter: vcan has no bus timing, and the PCAN-USB's own reception and
  timestamps are already covered by the recorder.
- The broker under memory pressure, or with the GPU/vision load of later phases.
- The real decoder: signals per frame, rates and topic layout are stand-ins.