04 — the lab

frames from the
running machine.

Not renders. Not mockups. These are real screenshots from the tooling I build and run — the perception HUDs, the flight telemetry, the simulator, the latency audits, and the pool where sim stops mattering. Tap any of them.

proof / engineering measured, not claimed from the running stack

five real decisions, with the numbers that settled them

01 architecture

Control costs vision nothing

Put the reflexes on hardware that cannot be interrupted, put the thinking where thinking is cheap, and be careful about the line between them.

The board runs a 500 Hz control loop on an ESP32 core — sensors, control, DShot. The Pi does nothing but see. They meet over one USB-C cable. The reason for the split is not tidiness; it is that a controller must never be late, and a neural net sometimes is.

The payoff shows up as a number you can read off the running system: when vision spikes, control does not. The loop on the Pi is slow on purpose, and the vehicle is steady anyway, because the fast loop lives where it belongs.

Control costs vision nothing — that’s the srot/Hailo separation doing its job, measured.

// source — mongla — srot/Hailo split, live telemetry

reflex-split ● live

control costs vision nothing — the srot/Hailo separation,
measured: control loop jitter unchanged under vision load.

02 the accelerator

Zero wasted decodes

The Hailo-8 has a scarce budget of TOPS. The whole camera path exists to make sure none of it is spent on a frame that no longer matters.

The camera only decodes a frame when the inference worker is idle. So the chip never spends its scarce compute on a frame that has already been superseded by a newer one. It is a simple rule — decode on demand, not on arrival — and it is the difference between a pipeline that looks busy and one that is fast.

The design does not merely claim this. The instrumentation says it, out loud, every ten seconds.

// source — mongla — chip-efficiency readout

778 inferences

11.5 ms frame age at infer-start (mean)

19.2 ms frame age (max)

0 decoded-but-never-inferred

03 the python problem

Nothing in the hot path holds the GIL

Python has one lock. On a vehicle, a stage that holds it while it works freezes everything else. So we audited every stage by hold × rate — the fraction of wall-time the process is frozen — instead of waiting for the next symptom.

The rule was applied systematically (tools/gil_audit.py), not chased one stall at a time. The verdict: every hot-path stage releases the lock. The only holder left is the viewer feed, which is negligible and off the mission path.

A parameter that is read, logged and then ignored is worse than one that is unsupported.

The cumulative result of this session’s work: ~28 ms of vision pipeline became 18 ms, ~6.5 ms of control-loop poll wait was removed, and detections roughly doubled — at about 30% less CPU.

// source — mongla — tools/gil_audit.py

28 → 18 ms vision pipeline

36.5 → 77 Hz detections

~30% less CPU

stage	duty	verdict
hailo run_async	78.3%	frees (was InferVStreams: held)
cv2.imdecode	32.0%	frees
letterbox	2.0%	frees
pyserial (srot link)	0.1%	frees
cv_bridge	0.1%	holds — viewer only, negligible

04 the honest ceiling

The sensor is the limit

When you optimise a pipeline, the goal is not “fast”. The goal is to make the software disappear as a source of delay — so that what is left is the camera itself. Verified on live hardware, both cameras, the verdict is the one you want.

Authority at p99 is 1.000: at the 99th percentile, the freshest frame the sensor could possibly provide is the frame the controller acts on. There is no software latency left to remove. The forward camera runs at 73.4 Hz, the downward at 27.6 Hz, and in both cases the bottleneck is the hardware, not the code.

The only honest verification that has ever worked here is looking at the frames.

// source — mongla — photon-to-command latency, live hardware

	forward	downward
rate	73.4 Hz	27.6 Hz
pipeline age med / p99	21.86 / 29.58 ms	46.94 / 52.44 ms
interval med	12.01 ms	32.01 ms
authority at p99	1.000	1.000
verdict	sensor is the limit	sensor is the limit

05 a first

XFeat on the Hailo-8 — a number nobody had published

Getting a state-of-the-art feature matcher to run on a neural accelerator that refused to parse it — and proving the port was exact, not approximate. No FPS figure for XFeat on the Hailo-8 appears to be published anywhere. So this is a new number.

Stock XFeat does not even parse for the Hailo-8: the compiler chokes on 71 Transposes, 72 Slices and 70 Unsqueezes, all 5-dimensional. The breakthrough was seeing that they are not 71 problems — they are ONE. They are `_unfold2d(x, ws=8)`, written with `Tensor.unfold`, which torch traces as a pile of 5-D slicing. That operation is space-to-depth, which torch already has as `pixel_unshuffle` and the chip supports natively.

The substitution is not an approximation, and the tool proves it rather than claiming it: `torch.equal` on random input, then the whole network against the stock graph — max abs diff 0.000e+00 across all three heads at both resolutions. It refuses to write otherwise, because a graph that compiles but computes something slightly different would surface later as a worse match rate and be blamed on quantisation.

A HEF that compiles is not a result.

The honest part is the best part. 701 FPS on the chip is 23× the Pi’s CPU — but only the backbone moved. Keypoint decoding, NMS and the matcher stay in numpy, and the match is the expensive half. So the real win is not the speed; it is that the bottleneck moved from the detector to the matcher, which reframes the next optimisation entirely. And the settings that shipped — top_k 1024, not 2048 — were chosen on real murky footage, not inherited: bank breadth beats keypoint density at equal cost.

// source — mongla — commits 908559e · 194c063 · 42466c8

701 FPS XFeat backbone on Hailo-8

23× vs the Pi CPU

0.000e+00 max abs diff vs stock graph

4/4 3/4 4/4 INT8 scores exactly what CPU scored

06 perception

Two cameras, one heading

Redundancy beats tuning. Instead of picking the one camera to trust, run both — a forward gate detector and a downward bin detector — and fuse their headings by how much each should be believed right now.

Each camera runs YOLO on CUDA at fp16, 640 px, and publishes to its own frame: a PID controller and a pure-CV estimator side by side, so a consumer can choose. The fusion weights the forward and downward headings (wf, wd) by confidence, not by a hardcoded preference.

It is the bumblebee doctrine in one screen: when you cannot pick a threshold, run both and let the consumer decide.

// source — duburi — dual-cam PID + pure-CV fusion

Two cameras, one heading — front + down, PID and pure-CV, fused fh1m

dual-cam ● live

ID:1 shaw_fish_gate 0.92   YOLO[front] cuda:0 fp16=True sz=640
ID:1 yellow_box_octagon 0.93   YOLO[down] cuda:0 fp16=True sz=640
FUSION (front + down)  fused heading +94.0°  (wf=0.36, wd=0.49)