There’s a commit in Yee, our code editor, that deletes a feature users liked. The feature was a little “save %” badge next to every model in the picker — gpt-4o saves 30%, gpt-4.1 saves 45%, and so on. It looked authoritative. People used it to pick models.
The numbers were made up.
Not maliciously — they were directional guesses, hardcoded before our measurement system existed, roughly calibrated to how each model behaved with our prompt and tooling layer. The kind of number every AI product ships. But once we built real per-language measurement, the tracker filed against that badge said it plainly, and I’ll quote our own source comment verbatim:
“a fabricated number sitting next to the real per-language measurement system undercuts the honest-instrument thesis. A model’s displayed savings is now MEASURED — the average of its adopted per-language-group deltas… or ‘unmeasured’ until an eval earns one.”
So today the picker shows one of two things: a measured number like ~38%, with a hover breakdown of exactly which language groups it was earned on — or a small dot. The dot means we haven’t measured this yet, so we won’t pretend. The tooltip on the dot doesn’t hedge either: “no measured savings yet — run the eval for a group to earn a number.”
The honest-instrument thesis
The idea underneath this is simple: an editor that routes your tokens is an instrument, and an instrument that lies about its own readings is worthless. Yee is bring-your-own-key — you pay your provider directly for every token our tooling spends. A savings percentage isn’t marketing copy in that world; it’s a claim about your money. Claims about your money get measured or they get deleted.
That principle shows up in small places once you commit to it:
- Model hints in the picker describe capability (“fast + cheap all-rounder,” “proven reasoning workhorse in this editor”) and are banned from carrying a savings percentage. The half-truth number is exactly what we dropped; capability curation is opinion, and labeled as such.
- Measured deltas are conservative by construction — they report the always-on savings levers, not the best-case peak from one flattering scenario.
- When a model hasn’t earned a number, its slot doesn’t inherit a sibling’s. The gpt-5.6 family shipped as “unmeasured” the day it landed, next to older models wearing real percentages. That looks worse on a screenshot. It’s true, though.
Why this is rare
Fabricated-but-plausible numbers are the water AI tooling swims in. Every landing page has a benchmark bar chart; every “2× faster” claim has a methodology nobody can find. The incentive is obvious — the number closes the sale, and the customer can’t audit it.
BYOK flips that incentive. Our users see their provider bill. If we claim 40% savings and their invoice says otherwise, we didn’t just lose a customer — we proved our instrument lies. The only durable move is the boring one: publish what you measured, on which stacks, and say “unmeasured” everywhere else.
This is also why our language support is graded in public, with the same rule — no tier claim without a bench behind it — and why we wrote up where agent token spend actually goes instead of hand-waving about efficiency.
The takeaway for your own tools
If you’re building anything that spends someone else’s tokens — an agent, an internal copilot, a data pipeline with an LLM in the loop — steal the rule, not the code:
- Separate measured from asserted, visibly. A dot that means “we don’t know yet” builds more trust than a plausible number ever will.
- Make the measurement earn its way in. A number appears when an eval produces it, never before.
- Delete the legacy guesses. Keeping them “for continuity” next to real measurements poisons both.
We deleted ours. The picker looks slightly less impressive and considerably more true — which, it turns out, is the trade the whole product is built on.