Code generation: a model at 1/271st the price of the best scorer, at 97.8% of its quality.

The last 7.4 points of rag answer quality cost 16×.

Potion Research

On Potion's measured rag-answer suite, one model scored 0.981 at $0.0937 per thousand requests while another scored 0.907 at $0.0058. Both figures come from the same held-out rag answer suite, 54 scored items, measured the same way — so the comparison is like for like.

The premium buys 7.4 quality points on this suite. Whether that is worth 16× the price depends on what a failure costs you: at high volume with a cheap retry, it usually is not; on work that ships unreviewed, it can be.

Someone is paying the difference. If that is you, the question worth asking is which of these two numbers your workload actually needs — and this is the measurement that answers it.

1 measurement cycle ran in the last 24 hours, 1 newly listed model was measured, no recipe reached a frontier.

The current numbers live on the measured answers; the method is public.

The last 7.4 points of rag answer quality cost 16×. · Frontier Notes