The Cheap Model Turned Out to Be the Expensive One

I did the math on Haiku, and it looked like an easy win.

I run a pipeline I call Great Minds. It's a background program that takes an idea and hands it to a series of AI agents. One group argues about the idea, another makes a plan, another writes the code and another checks the work. At the time every one of those agents ran on the same model. Haiku is Anthropic's smallest and cheapest model. The build step writes the most text, so it costs the most, and moving just that step to Haiku looked like it would cut a big chunk off the bill.

So we tried it. We were building plugins for Emdash, a newer website platform. Every build came back with more than 100 violations.

Here's what a violation means. The pipeline runs an automatic check on new code that looks for patterns we've banned because we know they break on this platform. On Haiku, every build tripped that check more than a hundred times. The model wasn't being sloppy, exactly. It was writing confident, neat code for parts of Emdash that don't exist. It made up the platform's tools and then used them correctly.

That's when I realized I'd priced the wrong thing.

A failed build doesn't stop at one failure. The checking step blocks it. A fixing step tries to repair it. Then the check runs again, finds the next layer of problems, and blocks it again. When we added it up, we estimated four to six of those rounds for each made-up build. Every round means rereading the code, rewriting it and checking it again, and every round costs tokens, which are the units these models charge by. So the cheap build step ended up as the most expensive part of the whole pipeline. The money we saved writing the code went into fixing it several times over.

I'd already seen this from the other side. Before the Haiku test there was a plugin called EventDash: 3,600 lines of code written against an API that doesn't exist. (An API is the set of commands a platform gives you to work with it.) It had 121 lines of throw new Response(), 14 references to rc.user, and it reached for process.env, none of which works the way the code assumed. We didn't fix it. We threw it out and wrote 130 lines based on the real documentation, which we checked first. Those 130 lines worked.

The 3,600-line version wasn't cheap just because it was quick to produce. It cost us everything we spent trying to rescue it.

The same mistake, in the other direction

Around the same time I made the opposite mistake on a different project. I was thinking about upgrading to gpt-4.1-mini and estimated the cost from the token cap, which is the maximum length you let a model write in one reply. Assuming every reply hit the cap, the upgrade came out at about fifteen times what we were paying. That made me nervous.

But the model never got close to the cap. It finished on its own at around 5,000 tokens. The scary number described a worst case that never happened.

In one case I thought the cheap model would save money and it didn't. In the other I thought the better model would cost a fortune and it didn't. Both times the mistake was the same: I priced the wrong unit. I looked at the cost per token, or the cost at the cap, when what I actually pay for is code that passes its checks and ships. That's the unit that matters. You count how many tries it takes to get there, and the price per token only tells you what each try costs.

So the fix wasn't "always use the expensive model." We wrote the model choice into each step of the pipeline, so every step gets the model suited to it, chosen on purpose. Right now Sonnet runs the debate, the plan, the build and the checking. We'll change that when the numbers change. But each step has to earn its model by what it actually ships, not by what it costs per word.

If you want to know whether a model is cheap, don't look at its price list. Count how many rounds of fixing it takes before something actually works.

---

Seth Shoultes builds at garagedoorscience.com and writes here when the building produces something worth saying.