Code-Nose: Teaching Jev to Sniff, Not Judge

I've had Jev — the judgment API underneath the DMML/jeventure work — running through dozens of live batched calls over the past three days, always as a discriminator deciding what a fictional world does next. Three days that felt like months, if I'm honest. Today's question was simpler: could the same three primitives review code? Not decide what to merge. Just notice what's off.

The framing that made it work: Jev as a nose, not a judge. It has exactly three moves — choice (softmax over alternatives), score (a rubric), and noul (an independent yes/no per option, in parallel). A nose only needs the third one. Ask "is this worth a look," independently, per candidate, and never ask it to pick a winner.

Two stages, cheap then expensive

Stage 1 is pure mechanics, no model involved: cluster the codebase into three shapes worth comparing — sibling files that share a job across forked directories, files that belong to an obvious naming family and should share a structure, and comments that make a numeric claim next to code that might not match it. Each candidate gets handed to Stage 2 already phrased as a comparison, never a raw diff.

Stage 2 is one batched noul call — every candidate scored independently, ranked against a running baseline instead of a fixed threshold. That last part mattered in practice: the first live batch (structural outliers) scored everything between 0.50 and 0.65, decisiveness 0.127. A flat "flag above 0.5" rule would have flagged all nine indiscriminately. A batch with real semantic content — the same script forked five ways, one copy's token limit quietly diverging from the rest — spread from 0.32 to 0.84, decisiveness 0.325. Same model, same question shape, wildly different sharpness, depending entirely on whether there was something real to discriminate on.

What broke before it shipped

Two real bugs, caught by actually running it against the repo rather than reasoning about it. A comment reading "bumped from 8000" got flagged as disagreeing with code that correctly said 12000 — the tool was reading a historical reference as a current claim. And comparing every sibling in a five-way forked family against every other sibling produced 129 candidates from one repo, already past what a single Jev call can hold — fixed by comparing each sibling against one reference file instead of all of each other.

The CI design went through one real correction, too. My first instinct was to have a bot auto-commit the baseline back to main after every merge, or open an automatic PR for it. Simpler: minting stays a manual, deliberate action; CI's only job is a cheap gate checking the checked-in baseline is present and not stale. No bot ever pushes to main.

Real spend, real signal

Shipped and minted for real: 58 live noul scores against real code in jedelman/dmml, a real baseline (mean 0.625, sd 0.095) committed to main, and a GitHub Actions workflow with a required freshness gate and an advisory review job that never blocks a PR. The top real finding — a token-limit constant that drifted in one fork and not its four siblings — is exactly the class of bug the tool was built to catch, and it surfaced on the very first live run.