The most expensive bug we ever had wasn’t a bug.
One of our internal apps is a web crawler — a background worker that fetches competitor pages for our SEO intelligence dashboard. Crawlers have a personality: they hit rate limits, collect 403s from defensive sites, retry with backoff, drop connections mid-read. That’s not failure; that’s Tuesday. The console output of a healthy crawler looks like a crime scene.
Now put an AI coding agent in front of that console.
The agent sees 403 Forbidden. Errors are for fixing, so it fixes: adjusts headers, adds retry logic that already exists, re-runs the crawler to verify. The crawler runs — and produces fresh 403s, because the sites it crawls didn’t stop defending themselves. New evidence of “failure.” New fixes. Re-run. Repeat. Every loop reads the whole scrollback, reasons hard about network behavior it cannot control, edits healthy code, and bills us for the privilege.
By the time a human looked up, the meter read roughly a million tokens. The incident is now immortalized as a comment in Yee’s source, next to the code that prevents it — “the exact loop that burned 1M tokens on dev3lop/intel.”
The root cause isn’t intelligence — it’s category error
The model wasn’t dumb. Given the frame “here is program output containing errors,” relentlessly pursuing fixes is correct behavior. The failure was ours: the harness handed a background service’s runtime chatter to an agent whose entire training says errors-in-console = broken code. No amount of model quality fixes a wrong frame; a smarter model just chases the 403s more ingeniously.
That reframing — the frame is the bug, not the model — is what produced the guardrails. Three shipped in Yee:
1. Apps get classified before their output gets interpreted. The preview system now distinguishes web apps from services — workers, crawlers, CLIs. A service’s runtime output is treated as expected behavior by default, and the agent’s context says so in words we mean, verbatim from the note it receives:
“Its console output (network 403s, retries, rate-limits, closed-connection races, queue churn) is EXPECTED RUNTIME BEHAVIOR of a live worker, NOT code bugs to fix — do not chase them with edits.”
Classification is evidence-based with a human override: if a “service” actually serves a page, one click reclassifies it, and if a forced “web app” never prints a URL, the runtime demotes it back — remembering the demotion so the mistake doesn’t loop either.
2. Framework noise never reaches the model. A filter strips the chatter that isn’t anyone’s bug — hot-reload announcements, devtools banners, Electron security boilerplate — before console output is ever packaged for the agent. The model can’t chase what it never sees, and you don’t pay for it to read boilerplate.
3. The fix loop is human-gated. Real errors don’t stream into the agent automatically. They surface to the person as a red ”⚠ fix errors” button; clicking it is what sends them. A human glancing at a console and deciding “yes, that’s worth model attention” costs zero tokens and catches exactly the category error that cost us a million — which is the whole thesis of our zero-token previews post: deterministic infrastructure for state, model spend reserved for judgment.
The general lesson for agent builders
Every agent harness has a version of this incident waiting. Anywhere your system feeds an agent observations of a live environment — logs, queues, monitors, network traces — you have an interpretation layer, whether you designed one or not. The default interpretation (“this text contains errors, errors mean broken code”) is wrong precisely for the systems that emit the most text.
The checklist we’d hand any team after this:
- Classify the source before sharing its output. A build’s stderr and a crawler’s stderr deserve opposite default framings.
- Filter known noise mechanically. Regex is cheaper than reasoning, by about six orders of magnitude.
- Put a human between observation and obsession. A one-click gate on “should the model see these errors?” converts runaway loops into non-events.
- Write the incident into the source. Our million-token comment isn’t shame — it’s the cheapest guardrail-preserving documentation we own. Nobody deletes the check with the price tag written next to it.
We measure and publish what our tooling saves. It’s only fair to publish what its absence once cost.