Every AI coding tool eventually makes a claim like “this cuts your token spend by X%.” We’ve made that claim too. Then we spent a month building the measurement infrastructure to check whether our own claims were true, and three of them weren’t — not because the underlying idea was bad, but because the evidence wasn’t.
This is a record of what broke, with the actual numbers, in case it’s useful to anyone else building or evaluating token-saving tooling.
1. n=3 is not evidence. It’s a coin flip wearing a lab coat.
Our first paid evaluation runs looked great. A CSS-focused prompt tweak came in at −60% cheaper at n=3. A Python variant showed −27% cheaper, also at n=3. Both looked like clean wins.
At n=5, both flipped sign: CSS went to +58% pricier, Python to +21% pricier.
Nothing about the tool changed between n=3 and n=5. The raw token counts on these cells swing 2–3x run to run — the kind of variance that makes a 3-sample confidence interval which happens not to cross zero look decisive when it’s actually noise dressed up as a result. If we’d shipped the n=3 numbers (and we nearly did — “see green on evals before we spend more” was the actual internal directive that week), we’d have published a false claim in good faith.
Fix: a hard floor of n≥5 (SIG_MIN_STEPS) before any result is allowed to be labeled significant, full stop. This one rule alone killed every false positive our early runs produced — each one was either under the floor or had a confidence interval that spanned zero once you looked honestly.
2. A “small effect, few samples” test almost killed our one real result
Once we had the n≥5 floor, we added a power check: could this sample size even detect a 25% effect if one were there? The test flagged our best result — a genuine, mechanism-backed win — as underpowered and tried to veto it.
The result it almost killed: a weak-model rescue lever measured at −53% cheaper, CI [−58%, −39%], on n=5. A 25%-effect power test needs roughly 7 paired samples to certify itself. We had 5. But we hadn’t measured a 25% effect — we’d measured a 53% one, with a confidence interval nowhere near zero. The power test was answering the wrong question.
Fix: the bootstrap confidence interval already prices in variance and sample size — a decisive, large-effect CI is significant even when the sample is underpowered for detecting a smaller effect. We now use power as an advisory flag (“this small effect might need more pairs before you trust its magnitude”), not a gate that overrides a CI that has already separated from zero.
Method, for anyone replicating this: stratified log-ratio bootstrap on the paired token delta, resampled within each test scenario so one large pairing can’t dominate a small one, on the log scale (the natural scale for a multiplicative token effect). We gate on total billed tokens — never a reasoning-stripped or output-only sub-metric the actual invoice wouldn’t back up.
3. Averaging across model tiers hides two real, opposite effects
Once the lever above cleared the bar, we checked it per model instead of pooling all models into one number. Good thing we did:
- gpt-4o: −53% significantly cheaper (the rich version of the prompt genuinely rescues a weaker model)
- opus-4.8: +20% robustly pricier, CI [19%, 20%] (the same rich prompt is pure overhead for a model that already does the task correctly without the extra scaffolding)
A pooled average across both models would have landed somewhere near zero and told us nothing — or worse, told us the lever “roughly doesn’t matter,” when in fact it’s a strong win for one tier and a strong loss for another. Token-saving is not a property of a tool. It’s a property of (tool × model capability). Any evidence standard that reports a single blended percentage across model tiers is structurally unable to see this.
4. A statistically clean result can still be measuring nothing that shipped
This one isn’t about statistics at all. We had per-language prompt tuning that our own eval harness reported as done — distinct, tuned prompts per language group, properly measured. Then it turned out the production configuration table (SHIPPED_PACKS) was empty. Every model, every language, was silently falling through to one identical global default. The plumbing for per-language prompts existed; the actual shipped prompts didn’t. The eval numbers were real measurements of a code path that no user ever hit.
Separately, around the same time, we found our own token-usage telemetry had been silently reading near-zero for months — not because nothing was happening, but because the API gateway wasn’t forwarding the usage field on the default model path, so every downstream meter, per-conversation total, and historical chart was quietly no-op’ing on an empty value. During that window, any savings claim we’d made, at any sample size, with any statistical rigor, would have been measuring a broken pipe, not the tool.
Takeaway: statistical rigor (n≥5, bootstrap CIs, per-tier stratification) is necessary but not sufficient. Before trusting a delta, verify the harness is actually instrumented against the code path that real users hit — not a shadow of it that only exists in the eval config.
The checklist we now hold ourselves to
Before we let any token-saving number leave the building:
- n≥5 minimum, no exceptions, regardless of how clean a smaller sample looks.
- Bootstrap CI on the log-ratio of paired billed-token totals — not a normal-approximation interval on a metric that isn’t normally distributed, and not a reasoning-stripped or output-only proxy.
- Power is advisory, not a veto — a decisive CI beats a power test tuned to a different effect size.
- Report per model tier, never pooled — a single blended percentage can hide two real, opposite effects.
- Verify the eval is scoring the shipped path, not a config that only exists in the harness.
None of this is exotic statistics. It’s the standard you’d apply to any A/B test in any other domain. The only reason it’s worth writing down is that “context window screenshot” and “looks smaller in this one run” are apparently still treated as evidence in a lot of agent-tooling marketing — including, on a few occasions this year, our own early drafts.
If you’re building or evaluating a token-saving tool and want to compare notes on the measurement infrastructure, reach out.