📊The meter

This is the page that changes how you think about agents, and it is mostly arithmetic.

Everything up to now has treated an iteration as free. It is not. Every iteration is a full API call carrying a conversation that keeps growing, and the shape of that growth is the reason projects #4 and #5 exist at all.

Part 1 · Where the money actually goes

Here are the real numbers from one three-iteration run — "roll 3d20, then put that many cookies in the jar" — before any caching was added.

Seven and a half thousand input tokens to roll some dice and count some cookies.

Now look at why it is that much

Read the column again. Iteration 2 is not 138 tokens. It is 2,513 tokens — because it contains everything iteration 1 contained, plus a bit.

Which makes cost quadratic, not linear

If each iteration adds roughly the same amount of new material, then iteration n costs roughly n units, and the run costs 1 + 2 + 3 + … + n.

That sum is n(n+1)/2 — the area under a growing line, not its length.

A five-step task does not cost five times a one-step task. It costs more.

Part 2 · One breakpoint, 47% off

Look at what is in those tokens. The system prompt and the eight tool definitions never change. Only the conversation on the end of them grows.

Most of that bill is the same bytes, three times.

Prompt caching lets you mark a prefix of the request as reusable. The API stores it; subsequent requests that begin with byte-identical content read from the cache instead of processing it again.

The loop puts a single cache_control breakpoint on the system block — which, because the API renders tools → system → messages in that order, covers the tool definitions too.

The same run, after:

Same tokens sent either way. A cache read costs roughly a tenth of a fresh token, so the full-price equivalent drops from 7,540 to about 4,004.

Part 3 · The two ways to break it

Prompt caching is a prefix match. That single fact generates both failure modes.

Break #1 — anything variable in the prefix

Interpolate a timestamp, a session id, a username or a random request id into the system prompt, and every request has a different prefix. Every hit silently becomes a miss.

// ❌ every request is a cache miss, forever
system: `You are a helpful assistant. The time is ${new Date().toISOString()}.`
 
// ✅ constant prefix; put the variable part in the messages
system: `You are a helpful assistant.`

Break #2 — a server that shuffles its tool list

This one is not your fault, and you can still be the victim of it.

Your tool definitions sit near the front of every request. They come from tools/list. If a server returns its tools in a different order on different calls — because it iterated a hash map, or did a SELECT with no ORDER BY — then your prefix changes and your cache breaks.

The specification calls this out directly:

Servers SHOULD return tools in a deterministic order […] Deterministic ordering enables clients to reliably cache the tool list and improves LLM prompt cache hit rates when tools are included in model context.

Part 4 · Reading the meter correctly

This is the part that looks like a measurement bug the first time you see it.

Your input tokens fall from 2,375 to 94 and your instinct is that the counter is broken.

It isn't. input_tokens is only the uncached remainder — what you paid full price for. The real prompt size is:

input_tokens  +  cache_creation_input_tokens  +  cache_read_input_tokens

Part 5 · Three seatbelts, and the one people skip

All in lib/agent-loop.ts:

1. 1

   MAX_ITERATIONS = 10

   A hard stop that says so loudly rather than quietly spending money.

   An unbounded tool-calling loop is not a hang — a normal runaway loop burns CPU you have already paid for; this one makes a paid API call every time round, with a conversation that is larger every time.

2. 2

   Explicit stop_reason handling

   end_turn, max_tokens, refusal, pause_turn each get their own branch. A loop that only asks "is it tool_use?" is a loop that can spin on a case you did not think about.

3. 3

   Token counts in the UI, per iteration

   Partly a feature. Mostly a seatbelt.

   This is the one people skip, and it is the one that makes the other two unnecessary — because you notice the shape of the curve long before you hit the cap.

You cannot manage a cost you cannot see.

By project #5 this has grown into a ledger with a row for every draw — including the refused ones, because a ledger with no refusals in it is indistinguishable from a gate that is switched off.

What you now know

• Every iteration re-sends the whole conversation, so cost is quadratic in iterations, not linear.
• One cache_control breakpoint on the system block cut a real run's full-price input by 47%.
• Caching is a prefix match: a timestamp in the system prompt, or a server that shuffles its tool list, silently disables it with no error.
• input_tokens is only the uncached remainder — real size is input + cache_creation + cache_read, and reporting one of the three invites a false comparison.
• Cap the loop, branch on every stop_reason, and put the meter on the screen.

Next: the twelve stages of building this, and the VERCEL_URL trap that survives a green deployment.