🪤The measurement trap

This is the most useful page in this section, and it is not about agents at all. It is about a bug in a number.

If you take one thing from project #4, take this rather than sub-agents.

Part 1 · What the README originally said

The crew costs 1.7× more at 60 jars — delegation is pure overhead at this size.

Confident. Specific. Backed by a script that printed real token counts from real runs against a real API. Written up, published, and believed.

Both the number and the conclusion were wrong.

Part 2 · Where 1.7× came from

The script divided one mode's total token spend by the other's.

// ❌ what it did
const ratio = crew.usage.total / single.usage.total;

Which would be fine if both modes had done the same amount of work.

They had not.

The eval suite is not symmetric

Some cases only exist in crew mode. You cannot measure "the crew brings every destructive call to one approval" with delegation switched off — there is no crew to bring anything.

Part 3 · Why it survived review

This is the part worth sitting with, because it is what makes this class of bug dangerous rather than merely annoying.

Nothing in that report was fabricated, rounded badly, or misread. Every token count came from a real API response. Every run genuinely happened. Every row in the table was correct. The arithmetic was correct.

Every individual number was right. The division was the lie.

There is nothing to catch by double-checking the numbers, because the numbers are fine. The error lives in the relationship between two of them, and that relationship is not written down anywhere on the page.

Part 4 · The fix, and the corrected result

Track tokens per observation group, and compare only the groups that both modes actually executed.

const sharedGroups = [...single.usageByGroup.keys()]
  .filter((k) => crew.usageByGroup.has(k));

  case                   ONE AGENT     A CREW
pantry-sweep            100% 3/3    100% 3/3
pantry-no-overkill      100% 3/3    100% 3/3

COST — on the run type(s) BOTH modes did
  ONE AGENT      198,020 tokens
  A CREW         166,509 tokens

→ No meaningful difference at 60 jars, either way.

Note the header on that cost block — on the run type(s) BOTH modes did.

Part 5 · The general shape

Any comparison between two configurations has to check that they did the same work, not just that they both ran.

This is an A/B test with unequal traffic, which is one of the oldest measurement errors there is. It arrives in agent work wearing an unfamiliar costume: some eval cases only make sense in one mode.

Once you know the shape, you start seeing it:

Part 6 · And then the sting

Project #4 documented this. Wrote the gotcha. Explained the fix. Built usageByGroup to do it properly. Corrected the README table.

And never wired usageByGroup up to the headline ratio.

usageByGroup was populated on line 185 and stored on the results object. The verdict on line 313 read results[n].usage — the raw totals — ignoring it entirely.

Project #5 caught it, still live, in scripts/08-compare.ts. The script had never been run in that repo. Run once, it reported:

  ANSWER QUALITY : one agent 100%   vs   crew 100%   (+0 points)
COST           : the crew cost 2.01x what one agent cost
→ Same answer, 2.01x the price. At 60 jars delegation is pure overhead.

A conclusion project #4's own corrected README already contradicts. Fixed there: 2.01× became 1.08×.

The only evidence that a fix works is the output of the fixed program.

Writing "which turns 1.87× into 0.84×" without re-running is how a corrected bug stays shipped.

Part 7 · Three habits, all cheap

1. 1

   Print the denominator next to the ratio

   Not 1.7× but 1.7× — 6 crew runs vs 3 single runs.

   A ratio without its inputs is an assertion. With them, it is a self-auditing figure — and this one would have been caught by whoever read it first.

2. 2

   Make the comparison refuse rather than guess

   If the two arms did not run the same cases, the right output is:

   copy

   cannot compare: crew ran 2 cases single mode did not

   …not a number with a caveat somewhere below it. A tool that cannot produce a misleading number is better than one that produces a caveated one, because the caveat gets dropped when the number gets quoted.

3. 3

   Re-run the check against the code, not the changelog

   The only reason project #5 found this is that it went and looked at what scripts/08-compare.ts actually did, rather than reading project #4's appendix and believing it.

   Inheriting a script is inheriting a claim. Run it once before you trust it.

What you now know

• The headline said 1.7× and was wrong, because the two modes ran different numbers of runs.
• Every individual number was accurate — the division was the lie, which is why review did not catch it.
• The fix is to compare only shared observation groups, and to print the denominator in the output.
• This is an A/B test with unequal traffic, and the same shape appears whenever two arms of a comparison do different amounts of work.
• A documented fix is not a fix. Project #4's correction existed in prose, in a comment and in a data structure, and never in the code path that printed the number.