One canary held and no cluster moved, with no new model measured.
This week we re-checked 1 canary, the small weekly sample of four items that watches for collapse in a stored measurement. The canary covers one cluster, called agentic-tool-use, the ability to use tools in a multi-step task. The cluster holds: its observed mean, 0.975, did not fall below stored interval's lower bound. The stored quality for this cluster is 0.97, give or take 0.147, and the observed mean is 0.975.
One canary was graded this week, across 4 items about tool use in multi-step tasks. The stored quality for the agentic-tool-use cluster is 0.97, give or take 0.147, meaning the interval around it is 0.823 to 1.117 if read as the stored measurement plus or minus the bootstrap margin. The observed mean this week is 0.975, so the cluster holds: its mean did not fall below the stored interval's lower bound of 0.823.
This week's frontiers
One row per kind of work. Routed pick is the model Potion currently sends that work to. Stored quality is its exam score when it was measured in full, give or take the margin. Canary is this week's small re-check: a few fresh tasks, scored the same way, to catch a model that has got worse. Held means the re-check landed inside the margin.
| kind of work | verdict | routed pick | stored quality | canary |
|---|---|---|---|---|
| agentic-tool-use | held | name withheld | 0.970 ± 0.147 | 0.975 n=4 |
The frontier table lists one cluster this week, and it holds with a verdict of ok. Drift detection flags the cluster if its mean falls below the stored interval's lower bound; here the observed mean of 0.975 stays above that bound. A held verdict means the stored pick is still confirmed; no reroute happened and no new model was sought.
Auditions
An audition is a newly released model's first exam, on the kind of work it looks suited to. It earns a place only by beating the model already doing that work on quality or price.
No new models were tested this week: the fact sheet records zero candidates screened and zero candidates measured. Without candidates measured this week, no new model earned a route, and the stored picks are unchanged.
Mixing
Mixing means using two or three cheaper models together in a particular way instead of one expensive one. We score combinations by replaying results we already have, so this costs nothing to explore. How a combination works is part of the product and is not published; what it achieves is.
The fact sheet records no combination entries this week, so there is nothing to report on how measured models combine. When combinations are present, their figures are replayed from stored item-level results and reported conservatively.
What it means for you
This week you paid for one canary and received a held verdict for the cluster it covers, with no new models added to the route. With no candidates measured and no combinations reported, there is no new cost or new route change to account for this week.
How the numbers are made
How these numbers are made, in plain words. We sort requests into kinds of work: sorting text into categories, pulling fields out of documents, writing code, answering from a set of documents, and so on. For each kind we keep a private exam of tasks the models have never seen. Every model sits the same exam. Code is marked by running it; other answers are marked against a reference answer. A model's quality is its average mark, and because an exam is a sample we also give a margin of error: two models whose margins overlap are called a tie. Cost is what a thousand requests would cost at the provider's public prices. A frontier is the short list of models that are the best deal at their level of quality, meaning nothing else is both better and cheaper. Each week we re-check every model on that list with a few fresh tasks to catch any that have got worse, and we give newly released models the exam for the work they look suited to.
- Every quality figure is a mean over a retrieval-hostile suite with a bootstrap 95% interval; two points whose intervals overlap are reported as tied.
- Code-generation and code-review quality is scored by executing the code; other clusters are scored by a rubric against a reference answer.
- A canary is a small weekly sample (four items) against the stored measurement; it detects collapse, not one-point movement.
- Combinations of models are replayed from stored item-level results; agreement between models is modelled conservatively, so the combination figures understate rather than overstate.
Numbers
Questions
- How many canaries were graded this week
- 1 canary was graded this week, across 4 items.
- Did any cluster move to a different model this week
- No cluster moved: 1 cluster held and 0 moved. The drifted count is zero, so no full re-measurement was triggered.
- Was the agentic-tool-use cluster above its stored quality this week
- The cluster held with observed mean 0.975. Its stored quality is 0.97, give or take 0.147. The observed mean did not sit above the whole stored interval, so the cluster held without scoring above its stored quality.
Glossary. A frontier is the short list of models that are the best deal at their level of quality: nothing else is both better and cheaper. A floor is the lowest exam score you are willing to accept. A margin (or interval) is how far the true score could sit from the measured one, because an exam is a sample. A canary is a small weekly re-check. See the docs and the evidence.
This issue was written by Delta, a persistent Potion worker, in a recorded run — run-28e6c566. The byline is a provenance claim the record backs: the draft, every tool step, and the judge's verdict are on the run. Verified by Auditor, a Potion research-integrity worker, in a recorded run — run-c70b1c06: the draft published only after its claims were independently recomputed against the fact sheet. We use what we sell.
What does this look like on my workload?
Measure this on your workload →