Eight of ten frontiers held; classification and extraction drifted and are flagged for full re-measurement.

We graded 40 items across 10 canaries, spending $0.91, and held 8 of 10 clusters while 2 clusters drifted and 0 were inconclusive. Of the 8 held clusters, 4 scored above their stored interval — code-review hit 1, give or take nothing reported, against a stored quality of 0.869, give or take 0.089, and rewrite-edit hit 0.975 against a stored quality of 0.866, give or take 0.046. The 2 drifted clusters both sit in the structured output family: classification observed 0.75 against a stored 0.976, give or take 0.033, and extraction observed 0.75 against a stored 0.962, give or take 0.017; both are flagged for full re-measurement and neither changed route.

This week's frontiers

One row per kind of work. Routed pick is the model Potion currently sends that work to. Stored quality is its exam score when it was measured in full, give or take the margin. Canary is this week's small re-check: a few fresh tasks, scored the same way, to catch a model that has got worse. Held means the re-check landed inside the margin.

Every frontier check this week was a canary of four items per cluster, so each verdict rests on a small sample and a one-sided collapse test, not a precise quality remeasurement. A held verdict means the canary mean did not fall below the stored interval's lower bound; it does not mean the cluster scored at or above its stored quality, and four held clusters actually scored above the stored interval entirely. The two drifted clusters, classification and extraction, are flagged for full re-measurement; their stored picks are no longer confirmed, and no reroute happened this week.

Auditions

An audition is a newly released model's first exam, on the kind of work it looks suited to. It earns a place only by beating the model already doing that work on quality or price.

We screened 322 candidates and measured 3 of them, all in the small/cheap lane for the classification cluster, for $0.91 total. The three measured aliases were or-l3-lunaris-8b, or-nex-n2-mini, and or-gpt-oss-20b; none beat the incumbent, so classification kept its stored pick. Free-tier listings were excluded from auditions because their quality is not stable enough to measure.

• or-l3-lunaris-8b · small/cheap · classification · did not beat the incumbent
• or-nex-n2-mini · small/cheap · classification · did not beat the incumbent
• or-gpt-oss-20b · small/cheap · classification · did not beat the incumbent

Mixing

Mixing means using two or three cheaper models together in a particular way instead of one expensive one. We score combinations by replaying results we already have, so this costs nothing to explore. How a combination works is part of the product and is not published; what it achieves is.

For code work, a combination of measured models was more than 8× cheaper than the best single model and scored 1 on average across 90 items, matching the best single model exactly. For structured output work, a combination was between 4× and 8× cheaper and also scored 1 on average across 80 items, with no quality difference from the best single model. For summarization specifically, a combination was under 1.5× cheaper, scored 1 on average across 14 items, and saved 0.194 in cost against the best single model; this is the one combination where the exact cost saving is known rather than given as a band.

• code work · cheaper-and-as-good · quality 1.000 · more than 8× cheaper · n=90
• structured output work · cheaper-and-as-good · quality 1.000 · between 4× and 8× cheaper · n=80
• writing work · cheaper-and-as-good · quality 0.979 · between 2× and 4× cheaper · n=14
• retrieval work · cheaper-and-as-good · quality 0.980 · between 2× and 4× cheaper · n=50
• reasoning work · cheaper-and-as-good · quality 0.993 · between 1.5× and 2× cheaper · n=14
• summarization · cheaper-and-as-good · quality 1.000 · 19% cheaper · n=14
• code work · frontier-candidate · quality 0.995 · more expensive than the best single model · n=42
• writing work · frontier-candidate · quality 0.929 · more expensive than the best single model · n=14
• extraction · frontier-candidate · quality 0.987 · -507% cheaper · n=74

What it means for you

If you pay per request, two of your routes — classification and extraction — are not confirmed this week and should be re-measured before you rely on them at their old quality. Four held routes scored above their stored interval, so 'held' is a floor, not a promise of the old quality; do not read a held verdict as a quality guarantee. Combination routing can cut cost by more than 8× on code work and by 4× to 8× on structured output work without losing quality, but those figures are bands, not exact savings.

How the numbers are made

How these numbers are made, in plain words. We sort requests into kinds of work: sorting text into categories, pulling fields out of documents, writing code, answering from a set of documents, and so on. For each kind we keep a private exam of tasks the models have never seen. Every model sits the same exam. Code is marked by running it; other answers are marked against a reference answer. A model's quality is its average mark, and because an exam is a sample we also give a margin of error: two models whose margins overlap are called a tie. Cost is what a thousand requests would cost at the provider's public prices. A frontier is the short list of models that are the best deal at their level of quality, meaning nothing else is both better and cheaper. Each week we re-check every model on that list with a few fresh tasks to catch any that have got worse, and we give newly released models the exam for the work they look suited to.

• Every quality figure is a mean over a retrieval-hostile suite with a bootstrap 95% interval; two points whose intervals overlap are reported as tied.
• Code-generation and code-review quality is scored by executing the code; other clusters are scored by a rubric against a reference answer.
• A canary is a small weekly sample (four items) against the stored measurement; it detects collapse, not one-point movement.
• Combinations of models are replayed from stored item-level results; agreement between models is modelled conservatively, so the combination figures understate rather than overstate.
• Free-tier listings are excluded from auditions: their quality is not stable enough to measure.

Numbers

Questions

Which AI routes failed their weekly check this week?

Two routes failed: classification and extraction, both in the structured output family. Classification observed 0.75 against a stored quality of 0.976, give or take 0.033; extraction observed 0.75 against a stored quality of 0.962, give or take 0.017. Both are flagged for full re-measurement and neither changed route.

What does it mean when a route is listed as held?

Held means the weekly canary mean did not fall below the stored interval's lower bound, so the route did not collapse. It does not mean the route scored inside its old interval; four held routes this week scored above the stored interval instead.

Can combining models cut cost without losing quality?

Yes, by family. Code work was more than 8× cheaper at quality 1 across 90 items; structured output work was between 4× and 8× cheaper at quality 1 across 80 items; summarization was under 1.5× cheaper with a measured saving of 0.194 across 14 items. These are bands for most families, not exact savings.

Glossary. A frontier is the short list of models that are the best deal at their level of quality: nothing else is both better and cheaper. A floor is the lowest exam score you are willing to accept. A margin (or interval) is how far the true score could sit from the measured one, because an exam is a sample. A canary is a small weekly re-check. See the docs and the evidence.