🆓The suite that costs nothing

Project #4's NEXT_STEP.md said to build this before anything else. It was right, and it changed how the whole project got built.

An eval case is two things, and only one of them costs money

   a PROMPT   "empty the cookie jar completely"        ← costs money
   a CHECK    "…and some call should have been gated"  ← a pure function

Every time you refactor, you want to re-ask the second question. Almost never the first.

And here is the thing that was already true and nobody had noticed: project #4 writes every loop event to trace_events, permanently, for the replay UI. So the model's decisions are already on disk. Re-running check() against a stored trace asks the same question about the same behaviour and costs zero.

  ✅ look-only                [single]  ✅ pantry-sweep         [crew]
✅ chain-dice-to-cookies    [single]  ✅ pantry-no-overkill   [crew]
✅ no-tools                 [single]  ✅ crew-delegates       [crew]
✅ gate-fires               [single]  ✅ crew-one-approval    [crew]
✅ picks-right-cipher       [single]  ✅ sampling-happens     [single]
✅ reads-history            [single]  ✅ sampling-attributed  [single]
                                      ✅ gate-refuses-expensive

SCORE: 100%   (13/13 scored)
Tokens spent by this command: 0

The one piece of real work

A stored trace is a tree. A live stream is flat.

lib/run-driver.ts deliberately writes a sub-agent's events to the child's trace, under the child's own sequence numbers — which is precisely what makes "expand this worker" a single query in the replay UI.

So rebuilding the live stream from storage means walking the tree and re-wrapping every child event on the way back up. Skip that and a delegated run replays as though the workers never called a tool — and the crew cases score zero for a reason that has nothing to do with the crew.

Two decisions that make it trustworthy

Both are about not lying, which is the only property that matters in a thing whose job is to tell you whether you broke something.

The extraction lives in one file

There are now two ways to watch an agent work: live from the loop, and replayed from Postgres. If each grew its own "turn events into a score sheet" logic they would drift.

And the failure would be vicious. The cheap suite would quietly score things differently from the expensive one, while both printed confident percentages. You would run the free one fifty times a day and it would be answering a slightly different question every time.

no data is not fail

A case with no stored tape scored nothing. And nothing must never round to zero.

So the counts are kept apart and named separately. On a fresh database, the checkpoint is supposed to say this:

  NOTHING TO SCORE. No stored run matched any case's prompt.
This is not a failure — it means the tapes don't exist yet.

That message is honest. It also hid a real bug for a while — the replay runner was built first, and nothing was being recorded yet, so a genuinely broken recording path produced the same reassuring sentence as an empty one.

What replay cannot answer

Stated up front, because a tool that oversells itself gets trusted where it shouldn't be.

You need both suites. But only one of them has to be asked fifty times a day while you refactor, and it is not the expensive one.