AI Model Comparison

Evaluation guide

LLM Benchmarks Explained: What Leaderboards Measure

MMLU-Pro, GPQA, SWE-bench and LMArena explained: what each measures, how leaderboards mislead, and the order to read them.

Verified 2026-11-22

A leaderboard is a compressed, lossy summary of a model's behaviour. It is useful for cutting a field of thirty models down to five, and dangerous if you stop there. Here is what the main public benchmarks actually measure, in what order to read them, and the four ways they mislead.

The benchmarks you will see on every leaderboard

MMLU-Pro — a harder successor to the original MMLU. Several thousand multiple-choice questions across dozens of academic subjects, with ten answer options instead of four. It measures broad knowledge and multi-step reasoning. It does not measure whether a model can do anything with that knowledge.

GPQA Diamond — graduate-level science questions written to be un-Googleable: a non-expert with web access scores around 30%, experts around 65%. It is the cleanest public signal for hard scientific reasoning. Because it is small, a two-point difference is noise.

SWE-bench Verified — real GitHub issues from popular Python repositories, solved end to end and graded by running the project's own tests. This is the closest thing to a coding benchmark that reflects work: the model has to read a codebase, find the bug and produce a patch that passes CI. It is Python-heavy, which is its main limitation.

AIME / competition math — a few dozen competition problems per year. Scores are high enough now that only perfect or near-perfect results distinguish the top models. Treat small differences as noise.

LMArena — human pairwise votes on blind responses, turned into an Elo-style rating. It measures what people prefer in a chat setting, which is not the same as what is correct. Style, length and formatting move the score.

How to read a leaderboard without being fooled

1. Compare within a benchmark, never across. A 78 on one benchmark and a 92 on another tell you nothing about which model is better. They are different tests with different ceilings.

2. Look at the gap to the next model, not the absolute score. A one-point lead is inside the noise of most of these tests. A ten-point lead, replicated across two benchmark families, is a real difference.

3. Check the date and the version. Models get silently updated, benchmarks get revised, and a chart screenshotted eighteen months ago is archaeology. Every score is a claim about a specific model version on a specific date.

4. Ask who ran the test. Provider-published numbers are usually correct but selected: they run the benchmark on their best configuration, with the best prompt, and report the best of several runs. Independent leaderboards are better for comparison, worse for freshness.

The four ways benchmarks mislead

Contamination. If a benchmark's questions end up in a training set, the score measures recall, not reasoning. Every major benchmark has had contamination concerns. Hard, recently written tests are safer than old, widely published ones.

Prompt sensitivity. The same model can swing several points on the same benchmark depending on the prompt template, whether it is allowed to reason before answering, and how the answers are parsed. A leaderboard that does not publish its harness is telling you less than it appears to.

The average hides the shape. A model that is excellent at code and mediocre at writing has the same average as one that is mediocre at both. If your workload is 90% code, the average is the wrong number to optimise.

Chatting is not building. Arena-style preference scores reward answers that read well. Production work rewards answers that are correct, structured, and cheap to verify. These correlate, but only loosely.

The order we recommend

  1. Filter on hard numbers first. Price per million output tokens and context window decide the shortlist. That is what the comparison table and the model index are for.
  2. Then look at one reasoning benchmark and one coding benchmark. GPQA or MMLU-Pro for reasoning; SWE-bench Verified if you build software. Ignore the rest until the shortlist is five models.
  3. Then run your own evaluation. Fifty real tasks from your own product, graded by you or by a second model, on the shortlist. This is the step that decides, and it is the step almost nobody does.

Why this site does not publish a single score

Scores change with every model release and every benchmark revision, and a stale number is worse than no number. We link to the providers' own model pages and to public catalogues, and we date every price and specification we publish. When we publish a score, it will carry the model version, the benchmark version, the harness and the date — otherwise it is not worth writing down.