Round 7 closes NO — twice
A fine-tune that scored a clean win on every checkpoint turned out to be scored by the wrong instrument. Retracting it once wasn't enough — the second retraction is the one that actually holds.
The first version of this result looked like a win: a 2000-step checkpoint flipped the canary column from 12 inliers to a passing score. Sweeping every checkpoint instead of reporting the best one killed it — the same canary read 15, 10, 11, 10, 10, 8 across steps 2000 through 12000, against a stock value of 11. It degrades monotonically with training. The "good" number was a lucky draw, and reporting it alone would have been the single most flattering — and wrong — number in the project.
The cause traced to the training data: 60% of the set — 2,440 of 4,036 frames — was YOLO prediction output: footage with bounding boxes, class names and confidences already burned into the pixels. Hard edges and crisp glyphs are the easiest features in any frame and don't exist at inference time, so the descriptor spent its capacity learning furniture. Widened to 336 held-out pairs, median inliers read 19.0 on the untrained venue and 18.5 on the trained ones — near-identical, which is the signature of a data problem, not overfitting (overfitting would read better-on-trained, worse-on-held-out).
Round 7, take two
A clean 320×240 retrain then looked like the win instead — all eight checkpoints scored at or above stock on the evaluation metric, held-out median +17.0, +9.5, +11.0. Run through the actual shipped ONNX path, the same weights gave 2,052 total inliers against stock's 2,237 — an 8.3% loss, with the same 19-of-20 passing columns either way. No improvement. Round 7 closes NO, for real this time, and stock weights stayed.
A control arm that cannot reproduce its own recorded number is not a control arm.
The instrument was the defect. The fine-tune scorer used its own keypoint extraction — top-k on the heatmap, nearest-neighbour descriptor matching — not the shipped path's NMS, bilinear sampling and reliability weighting. It was a different algorithm wearing the same name, and the tell was visible from the first run: the scorer's own stock arm reported [210, 133, 222, 41] where the recorded table — produced by the shipped path — reads [189, 65, 134, 24]. Two different numbers for one unchanged model. I noticed it, blamed post-processing, and kept using that scorer for two training runs and four checkpoint sweeps before checking. That check costs one command. Not running it cost weeks.