Code generation: a model at 1/271st the price of the best scorer, at 97.8% of its quality.

The cheapest model that clears 0.99 on code gen costs 12× less than the dearest that does.

Potion Research

2 measured options clear 0.99 quality on code-gen; they range from $0.5537 to $6.85 per thousand requests. Both figures come from the same held-out code gen suite, 94 scored items, measured the same way — so the comparison is like for like.

Everything in that range clears the same bar on this suite, so the spread measures what a seller can charge rather than what a buyer receives. The only thing that moves the answer is whether your own quality bar is really the one measured here.

Someone is paying the difference. If that is you, the question worth asking is which of these two numbers your workload actually needs — and this is the measurement that answers it.

3 measurement cycles ran in the last 24 hours, 3 newly listed models were measured, no recipe reached a frontier.

The current numbers live on the measured answers; the method is public.

The cheapest model that clears 0.99 on code gen costs 12× less than the dearest that does. · Frontier Notes