The last 2.1 points of code gen quality cost 296×.

On Potion's measured code-gen suite, one model scored 1.000 at $6.85 per thousand requests while another scored 0.979 at $0.0232. Both figures come from the same held-out code gen suite, 94 scored items, measured the same way — so the comparison is like for like.

The premium buys 2.1 quality points on this suite. Whether that is worth 296× the price depends on what a failure costs you: at high volume with a cheap retry, it usually is not; on work that ships unreviewed, it can be.

Someone is paying the difference. If that is you, the question worth asking is which of these two numbers your workload actually needs — and this is the measurement that answers it.

1 measurement cycle ran in the last 24 hours, 1 newly listed model was measured, no recipe reached a frontier.

The current numbers live on the measured answers; the method is public.