Every model launch comes with its bar chart: “80.3% on SWE-Bench Pro!”, “state of the art on Terminal-Bench!”. Impressive. But what exam did the model actually sit, and who graded it? Good news: a benchmark is simply a standardized exam, and reading one is not rocket science.
The Olympic Games of models
Think of benchmarks as Olympic disciplines: same track, same rules, same stopwatch for every competitor. SWE-Bench Pro is the coding marathon (fix real bugs in real repositories); Terminal-Bench is the command-line relay; MRCR is the memory endurance event (find facts buried in a very long document).
Concretely, a benchmark is three things: a fixed set of exercises (hundreds to thousands), identical conditions for everyone, and automatic scoring. A script checks each answer (does the patched code pass the tests?) and outputs a percentage. That is what makes scores comparable, and podiums (the leaderboards) possible.
Reading a real scoreboard
Scores published in July 2026, for Anthropic’s Claude models and OpenAI’s GPT-5.6 family (Sol the flagship, Terra the mid-tier, Luna the light sprinter):
| Discipline | Fable 5 | Opus 4.8 | Sol | Terra | Luna |
|---|---|---|---|---|---|
| SWE-Bench Pro (fix real bugs) | 80.3% | 69.2% | 64.6% | — | — |
| Terminal-Bench 2.1 (command line) | 86.0% | — | 88.8% | 87.4% | 84.7% |
| MRCR (long-context recall) | — | — | 91.5% | 89.6% | 41.3% |
Three lessons. First, nobody wins every event: Fable 5 dominates the coding marathon by 15 points; Sol takes the terminal relay by 2.8. Second, when four models finish within 4 points of each other, price becomes the tiebreaker: Luna costs $1/$6 per million tokens against $10/$50 for Fable 5. Third, Luna’s 41.3% on MRCR shows why model families exist: the light sprinter collapses in the endurance event.
What the medals don’t say
- Contamination: if test questions leak into the training data, the athlete has trained on the exact course. Scores inflate.
- Saturation: when everyone scores 95%+, the discipline stops discriminating, so harder ones get invented (SWE-Bench begat SWE-Bench Pro).
- Your job is not an Olympic discipline: a 100 m medal doesn’t prove someone can move your furniture. To know whether a model handles your tickets, in your domain, you need your own private exams: evals.
In summary
| Question | Answer |
|---|---|
| What is a benchmark? | A standardized public exam: fixed questions, same conditions for all |
| Who grades? | A script, automatically; that’s what makes scores comparable |
| Is there one champion? | No: winners per discipline, and per euro |
| Classic traps | Contamination (leaked questions), saturation (everyone at 95%+) |
| What it doesn’t measure | Your use case; that’s the job of your own evals |
Benchmarks tell you who runs fast on a standard track; your evals tell you who does your job well. Read the first with curiosity, trust the second. And next time a launch chart flashes by, you’ll know exactly what the percentage means: it’s not rocket science.
Sources: Vellum, Claude Fable 5 benchmarks · Vellum, GPT-5.6 benchmarks · OpenAI, previewing GPT-5.6 Sol.
