What I check before trusting a benchmark
New models arrive with tables of benchmark scores, and the scores keep going up. They are useful signals, but a number on a leaderboard says less about a particular use case than it appears to. Before I take one seriously, I go through a short list.
Is it close to my task?
A model that excels at competition math may be ordinary at summarizing meeting notes. Benchmarks measure specific skills, and the transfer between them is weaker than the averages imply. The closer a benchmark is to the work you actually do, the more its score is worth.
Could the answers have leaked?
Popular benchmarks are published, discussed, and copied across the internet. Over time, some of their questions end up in training data. A model that has seen the test will score well on it without being better at the underlying skill. Newer or private test sets are more trustworthy for this reason.
How was it measured?
Scores depend on details: how many examples were shown in the prompt, whether the model was allowed to reason before answering, how answers were extracted and graded. Two numbers from different setups are often not comparable, even when they appear side by side in the same table.
Is the difference meaningful?
A one-point gap on a few hundred questions can be noise. Small differences at the top of a leaderboard rarely translate into anything you would notice in practice.
The test that matters
In the end the most informative benchmark is a small one you build yourself from real examples of your own work. It takes an afternoon, it cannot have leaked, and it measures exactly what you care about. Public scores tell you which models are worth trying. Your own examples tell you which one to use.