Published in English · Sources and limits included · Lire en français
When the metric becomes the target, it stops measuring what matters.
This is not a claim that models stopped improving. It is a testable operating hypothesis: as common metrics became optimisation targets, their scores became less reliable on their own as evidence for systems that must survive outside the test.
Evidence contract
What sources establish
Benchmark saturation, protocol sensitivity and contamination can weaken what a score predicts beyond its test.
What we infer
Once an evaluation is close to the optimisation path, a score should carry less decision authority without independent transfer evidence.
What remains to test
Whether this effect increased after 2023 across fields, organisations and production environments. The date is a hypothesis boundary, not a measured break.
01 · The claim
A benchmark is a compressed theory of the product.
Accuracy, pass rates, arena scores, task completion and LLM-judge ratings are observations, not the product. Each encodes what matters, which failures count and which trade-offs are acceptable.
The 2023 boundary used here is an editorial hypothesis, not a statistical break in the field. It marks the period after which shared evaluations appeared to become more central to training, selection and market narratives. The question is whether that proximity reduced transfer value; the cited studies establish mechanisms, not the date.
A published score can still be useful. It simply cannot carry the whole decision without evidence about exposure, evaluator, harness and the environment in which the system must operate.
02 · The target
Goodhart's law becomes a delivery risk.
Goodhart's law warns that pressure on a measure changes the behavior it observes.[1] In AI delivery, the proxy can improve while the workflow, permissions, recovery or economics remain unchanged.
Eval-first guidance from OpenAI and Anthropic remains useful. The failure begins only when the evaluation becomes a permanent contract rather than an instrument that contrary evidence can replace.
03 · Professional responsibility
Knowing compliance with a dead measure.
An imperfect metric used in good faith differs from one promoted after its relationship to the claimed capability becomes materially uncertain. This is a claim-discipline problem, not an allegation of criminal fraud or a judgement on every researcher.
- Degraded proxy: saturation, contamination or weak validity is known.
- Familiar test: exposure, selection or repeated iteration narrows independence.
- Untested transfer: the gain has not met another distribution, evaluator or operating constraint.
- Oversized headline: the caveat narrows the evidence but not the public claim.
General authority requires an independent hold-out and a headline limited to what the evidence supports.
04 · The transfer gap
Once everyone knows the exam, the score carries less surprise.
A study of 3,765 benchmarks found rapid saturation across many tasks.[6] A separate study found that evaluation-format changes could move models by as many as eight leaderboard positions.[4] ConStat measured contamination through performance that failed to generalise to related samples.[5]
Together, these sources establish mechanisms that can weaken transfer. They do not establish a universal 2023 break. That broader claim remains a hypothesis requiring longitudinal, cross-domain validation.
Production adds judges the benchmark usually omits: permissions, recovery, traceability, latency, cost and operator workload. A score becomes decision evidence only when it predicts outcomes under those constraints.
05 · By the numbers
The observations are real; their scopes differ.
The benchmark count describes ecosystem saturation. The trajectory count is a reconstruction from published curves. The ranking movement describes protocol sensitivity. They support different parts of the mechanism and must not be added into one headline statistic.
ECOSYSTEM MAP HARNESS SENSITIVITY TRANSFER TEST
3,765 benchmarks up to 8 rank places performance on
947 tasks under small changes related samples
↓ ↓ ↓
saturation risk ranking instability contamination riskRead each number with its unit, population and source. None measures production value directly.
06 · Durable or local?
A durable pattern survives a judge its author did not design.
A successful demonstration becomes a reusable pattern only when four conditions hold: the evaluator is meaningfully independent; the result transfers across distributions or operators; failure remains possible and visible; and the effect persists after novelty, tuning and author attention are removed.
- Independent judge: someone outside the optimisation path controls part of the test.
- Transfer: the effect survives changed data, tools, permissions or users.
- Falsifiability: the team states what would retire the claim.
- Persistence: the outcome remains useful under sustained operation.
● evidenced◐ partial○ not established
Without those tests, the honest label is a local success story—not an approved pattern.
07 · An operating method
Keep the metric useful by limiting its authority.
A production evaluation should state the decision it informs, the environment it represents and the conditions that retire it. Pair the headline metric with cost, latency, authority and recovery measures, then keep an independent hold-out outside the optimiser’s view.
- Write the prediction. State what the metric should forecast outside the test.
- Add counter-metrics. Prevent quality gains from hiding cost, latency or control failures.
- Preserve an external judge. Use changed users, permissions, tools and time periods.
- Record negative evidence. Incidents, reversions and delayed outcomes stay in the ledger.
- Set retirement criteria. Saturated or decision-irrelevant metrics lose authority.
- Re-open after launch. Production evidence can revise the evaluation and the decision.
08 · Limits
What this argument does not establish.
- No measured 2023 break. The date is an editorial hypothesis boundary, not a field-wide estimate.
- No claim that capability stopped improving. Capability and the transfer value of an evaluation are different objects.
- No replacement of experiments by anecdotes. Production adds a judge; it does not remove experimental discipline.
- No single cause. Incentives, access, closed systems and limited operating experience remain competing explanations.
- No rejection of metrics. The objection is to treating one frozen proxy as the permanent definition of success.
A metric earns authority only while it predicts an outcome the organisation still values.
09 · Sources
External evidence used in this analysis.
10 · Prepare a discussion
Move from a score to a decision that can be tested.
A first conversation is not an automatic order or a promise of an audit. It frames the operating question before any intervention is proposed.
The score, protocol and failure.
One metric in use, how it is calculated and one real case it does not represent correctly.
Separate observation, decision and judge.
We identify what is established, what remains a hypothesis and which independent evidence would reduce uncertainty.
One bounded next test.
The decision to make, the validation boundary and the unknowns to preserve. Any engagement is discussed separately.