What is the best model for summarization?
As of September 19, 2026, the cheapest live-measured option within 90% of top quality on summarization is ling-3.0-flash at $0.0491 per 1,000 requests — 0.772 measured quality vs 0.824 at the top, with a 67× price spread across the measured frontier (frontier v6).
re-measured weekly · how these numbers are made
Summarization quality is subjective at the margins, which makes it exactly the workload where judge-based scores need calibration receipts. The measured suite anchors judging against references and calibrates judges against tasks with deterministic truth, publishing the correlation rather than asking for trust.
The measured frontier
Every row is a live measurement on Potion's held-out summarization suite — same items, same scoring, per option. Names that are part of the product are withheld; their numbers are not.
| option | vendor | measured quality | $ / 1K requests | p95 latency |
|---|---|---|---|---|
| ling-3.0-flash | inclusionai | 0.772 | $0.0491 | 13230 ms |
| mistral-small-3.2-24b-instruct | mistralai | 0.766 | $0.0713 | 10078 ms |
| gemini-2.5-flash-lite | 0.732 | $0.0815 | 914 ms | |
| Potion's routed pick (name withheld) | — | 0.778 | $0.0859 | 6896 ms |
| qwen3-235b-a22b-2507 | qwen | 0.820 | $0.1167 | 5583 ms |
| gemini-3.1-flash-lite | 0.796 | $0.2751 | 1383 ms | |
| gemma-4-31b-it | 0.802 | $0.3510 | 3868 ms | |
| mistral-medium-3.1 | mistralai | 0.798 | $0.3963 | 1349 ms |
| gemini-3.5-flash-lite | 0.790 | $0.4144 | 1280 ms | |
| claude-haiku-4.5 | anthropic | 0.800 | $1.12 | 3051 ms |
| gpt-4.1 | openai | 0.818 | $1.61 | 2343 ms |
| claude-sonnet-4.5 | anthropic | 0.824 | $3.31 | 5728 ms |
frontier v6 · measured 2026-09-19 · live provider calls only — simulated evidence never appears on this page
Head to head
ling-3.0-flash vs mistral-small-3.2-24b-instructgemini-2.5-flash-lite vs ling-3.0-flashling-3.0-flash vs qwen3-235b-a22b-2507gemini-3.1-flash-lite vs ling-3.0-flashgemma-4-31b-it vs ling-3.0-flashling-3.0-flash vs mistral-medium-3.1gemini-3.5-flash-lite vs ling-3.0-flashclaude-haiku-4.5 vs ling-3.0-flashgpt-4.1 vs ling-3.0-flashclaude-sonnet-4.5 vs ling-3.0-flashgemini-2.5-flash-lite vs mistral-small-3.2-24b-instructmistral-small-3.2-24b-instruct vs qwen3-235b-a22b-2507gemini-3.1-flash-lite vs mistral-small-3.2-24b-instructgemma-4-31b-it vs mistral-small-3.2-24b-instructmistral-medium-3.1 vs mistral-small-3.2-24b-instructgemini-3.5-flash-lite vs mistral-small-3.2-24b-instructclaude-haiku-4.5 vs mistral-small-3.2-24b-instructgpt-4.1 vs mistral-small-3.2-24b-instructclaude-sonnet-4.5 vs mistral-small-3.2-24b-instructgemini-2.5-flash-lite vs qwen3-235b-a22b-2507gemini-2.5-flash-lite vs gemini-3.1-flash-litegemini-2.5-flash-lite vs gemma-4-31b-itgemini-2.5-flash-lite vs mistral-medium-3.1gemini-2.5-flash-lite vs gemini-3.5-flash-liteclaude-haiku-4.5 vs gemini-2.5-flash-litegemini-2.5-flash-lite vs gpt-4.1claude-sonnet-4.5 vs gemini-2.5-flash-litegemini-3.1-flash-lite vs qwen3-235b-a22b-2507gemma-4-31b-it vs qwen3-235b-a22b-2507mistral-medium-3.1 vs qwen3-235b-a22b-2507gemini-3.5-flash-lite vs qwen3-235b-a22b-2507claude-haiku-4.5 vs qwen3-235b-a22b-2507gpt-4.1 vs qwen3-235b-a22b-2507claude-sonnet-4.5 vs qwen3-235b-a22b-2507gemini-3.1-flash-lite vs gemma-4-31b-itgemini-3.1-flash-lite vs mistral-medium-3.1gemini-3.1-flash-lite vs gemini-3.5-flash-liteclaude-haiku-4.5 vs gemini-3.1-flash-litegemini-3.1-flash-lite vs gpt-4.1claude-sonnet-4.5 vs gemini-3.1-flash-litegemma-4-31b-it vs mistral-medium-3.1gemini-3.5-flash-lite vs gemma-4-31b-itclaude-haiku-4.5 vs gemma-4-31b-itgemma-4-31b-it vs gpt-4.1claude-sonnet-4.5 vs gemma-4-31b-itgemini-3.5-flash-lite vs mistral-medium-3.1claude-haiku-4.5 vs mistral-medium-3.1gpt-4.1 vs mistral-medium-3.1claude-sonnet-4.5 vs mistral-medium-3.1claude-haiku-4.5 vs gemini-3.5-flash-litegemini-3.5-flash-lite vs gpt-4.1claude-sonnet-4.5 vs gemini-3.5-flash-liteclaude-haiku-4.5 vs gpt-4.1claude-haiku-4.5 vs claude-sonnet-4.5claude-sonnet-4.5 vs gpt-4.1
Questions
Can summarization quality be measured at all?
Yes, carefully: reference-anchored judging (the judge sees a reference summary) calibrates near-perfectly against deterministic truth; reference-free judging does not, and is not used for these numbers.
Do I need a frontier model to summarize?
For routine document and thread summarization, measured mid-tier models are frequently within the interval of frontier ones. Long, dense, or high-stakes material shifts the answer — measure on your own traffic before trusting anyone’s general number.
What matters more, model or prompt?
Both, but the frontier is measured with each model given the same instruction under the same scoring — the ranking isolates the model. Your prompt can move absolute quality; it rarely reorders a wide measured gap.
Potion routes each request to the cheapest option measured at your quality bar — with a receipt on every answer and this page's evidence behind every pick. Get an API key or read the docs.