Book a 30-min call
← cd ../blogs
$ cat posts/chance-corrected-eval-metrics-llm-judge-validity.mdx

Chance-corrected agreement: your LLM judge is worse than your dashboard says

June 30, 2026 · ImmovableTech Team

  • Evaluation
  • Production AI

Your dashboard’s headline number is probably exact match

Go and look at how your LLM judge was validated. In almost every pipeline we have seen, including our own, the number that got written into the slide is the proportion of items where the judge’s verdict matched the human label. Exact match. Raw agreement. It is the default in the tooling, it is the headline metric in the benchmark papers, and it systematically overstates how much discrimination the judge actually has, because it pays the judge for agreement that guessing alone would produce.

The largest systematic evaluation of LLM judges to date, by the authors’ count, puts a number on the overstatement. Norman, Rivera and Hughes at UC Berkeley ran 21 judges from nine providers across MT-Bench, JudgeBench and RewardBench under three protocols — agreement, consistency and a bias audit — over 118 runs and roughly 541,000 individual judgments, with the frontier tier drawn from models released through April 2026. On MT-Bench, every single judge’s exact-match score exceeded its Cohen’s κ, by between 33.8 and 41.3 percentage points, with a cohort mean of 38.6pp. Not most judges. All 21, across every provider, price tier and generation.

The strongest judge in the cohort on chance-corrected agreement still showed a 33.8pp gap: exact match of 0.849 against a κ of 0.511. The paper’s own summary of what this means in practice is blunt: a judge reporting 85% agreement on MT-Bench has a κ of about 0.48.

What chance correction actually does

Cohen’s κ is the observed agreement discounted by the agreement you would expect from chance, rescaled so that chance-level performance sits at zero:

κ = (p_o − p_e) / (1 − p_e)

p_o is the proportion of items the two raters agree on — exactly what exact match reports. p_e is the agreement expected if each rater assigned labels independently at their own observed rate, estimated from the empirical marginal label frequencies of both raters. κ of 1 is perfect agreement, 0 is no better than chance and negative values mean the two raters agree less than independent guessing would predict.

The important part is where p_e comes from. It is not a uniform prior over the label space; it is built from how often each rater actually uses each label. A judge that has learned to mimic the human label distribution raises p_e and therefore faces a harsher correction, which is the correct behaviour: matching the base rates is not the same as discriminating between items. Krippendorff’s α coincides with κ in the two-rater nominal case and generalises to more raters and to ordinal scales, which is why the paper uses it for the multi-run consistency measurements.

The arithmetic is unforgiving on balanced label sets. MT-Bench presents pairwise comparisons with an A/B/Tie label space that is close to balanced, so p_e lands near a third, and an exact match in the 0.80 to 0.85 range corresponds to a κ near 0.48. That is a moderate level of agreement, not the near-perfect band the percentage implies.

Our own 92% is the worked example

We published this mistake. Our hallucination detection pipeline post reported 92% factuality accuracy, and that figure is raw agreement between the pipeline’s verdict and a human label across a four-way verdict space — confirmed, contradicted, partially supported, unverifiable — that is heavily skewed towards confirmed. We did not publish a chance-corrected figure alongside it.

On a skewed label set the raw number flatters you in a specific, quantifiable way: a system that did nothing but always predict the majority class would score well above zero, and that free credit is inside the 92%. We did not compute κ at the time, so we cannot tell you what it was. The correct thing to say about a number you never computed is that you do not know it, not that it was probably fine.

What made it worse is that 92% was never the number that justified the system. The false-positive reduction and the auto-publish rate were, and both are measured against a baseline rather than against chance. We led with the metric that sounded best rather than the metric that carried the argument.

Deflation belongs to the benchmark, not the judge

The size of the gap is not a judge-quality signal. It tracks the label distribution of whatever you validated on. Across the same 21 judges, the mean gap between exact match and κ was 38.6pp on MT-Bench’s balanced ternary labels, roughly 24pp on JudgeBench’s correctness-labelled pairs and roughly 10pp on RewardBench’s imbalanced chosen-versus-rejected binaries. Balanced label distributions raise the chance baseline, which widens the gap, exactly as Cohen’s correction predicts.

Two consequences follow that are easy to get wrong in opposite directions.

First, you cannot infer anything about a judge from the size of its deflation. A judge with a 10pp gap is not a better judge than one with a 38pp gap; it was probably just measured on a more skewed dataset.

Second, and for the same reason, κ is not a neutral scale you can carry between datasets. Because p_e depends on the marginals, the same judge can score a substantially different κ on two datasets it handles equally well. Compare judges within a benchmark. Do not compare a κ from your internal eval set against a κ from a published leaderboard and conclude anything.

There is a sharp illustration of marginal-dependence in the paper’s appendix. RewardBench’s standard generative loader places every chosen response in position A and every rejected one in position B, so the human label is identically “A” on every item. Under that ordering p_e collapses and κ degenerates to zero for every judge, regardless of how well it performs. The authors evaluate with per-item position randomisation instead, which restores a balanced ordering and produces κ values between 0.616 and 0.898. A metric reporting zero told you nothing about the judges and everything about the loader.

Comparing two judges on a compressed benchmark is measuring noise

Chance-correcting your headline number does not fix the second problem, and conflating the two is a mistake. Rank instability is separate from deflation and it does not go away when you switch metrics.

MT-Bench compresses all 21 judges into a κ band 13.5pp wide, from 0.376 to 0.511, with an average gap of 0.6pp between adjacent ranks. JudgeBench spreads the same 21 judges across 60.4pp, from 0.271 to 0.875 — about 4.5 times sharper discrimination on an identical population. When adjacent ranks are separated by six thousandths of a κ point, the ordering is close to arbitrary.

And the ranks do move. Eleven of the 21 judges shift by four or more positions across benchmarks, and only two hold a top-three position on all three. One judge sits fifth on MT-Bench and twentieth on JudgeBench. (The paper’s abstract summarises the largest move as 14 positions while its results section and rank table give 15; the discrepancy is unresolved in the text, and the direction of the finding does not depend on which is right.)

The driver is not only discriminative power but construct. MT-Bench labels aesthetic preference between two responses; JudgeBench labels objective correctness on mathematics, coding, writing and analysis; RewardBench labels chosen against rejected. These are different latent things. A judge validated on preference-style data can collapse on correctness-style data, and the generational improvements you might care about are invisible on the compressed benchmark: one provider’s flagship line moves from κ 0.309 to 0.487 to 0.606 across three generations on JudgeBench while sitting at 0.451, 0.451 and 0.457 on MT-Bench.

This is also the limit of what chance correction buys you. κ tells you how much of your agreement is real rather than free. It says nothing about whether the labels encode the thing your product actually cares about. If your eval set labels a helpfulness preference and your users are harmed by factual errors, a κ of 0.9 is a well-measured answer to the wrong question.

“95% consistent” is not evidence of validity

The most common sanity check we see — rerun the judge, confirm the score barely moves, declare it reliable — is the one the paper dismantles most directly. Test-retest reliability measures the stability of a judge’s outputs, not the correctness of its decision process. The two are close to orthogonal.

A judge that deterministically prefers whichever response is shown first will reproduce itself perfectly across runs while being maximally biased. That is not a thought experiment. Two production-deployed judges in the cohort recorded test-retest above 0.95 alongside position bias above 0.10: one at 0.992 test-retest with 0.192 position bias, the other at 0.988 with 0.125. The first of those also posted one of the three lowest JudgeBench κ values in the study, at 0.289. It is the most reproducible judge in the cohort and among the least valid, and reporting reproducibility alone would have hidden that completely.

Position bias across the cohort ranged from 0.002 to 0.192, spanning nearly two orders of magnitude, and it does not track price or capability cleanly — two models in the same family differed by a factor of 70. There is no substitute for measuring it on your own judge and your own rubric.

The measurement is cheap. Run every pairwise item in both orderings and count how often the judge picks whatever is in front of it:

def position_audit(judge, items):
    """Run each pair as AB and BA; report position bias and flip rate."""
    first_wins = decisive = flips = 0
    for item in items:
        ab = judge(item.a, item.b)  # "first" | "second" | "tie"
        ba = judge(item.b, item.a)
        for verdict in (ab, ba):
            if verdict != "tie":
                decisive += 1
                first_wins += verdict == "first"
        # Name the winning response so the two orderings can be compared.
        ab_winner = {"first": "a", "second": "b", "tie": "tie"}[ab]
        ba_winner = {"first": "b", "second": "a", "tie": "tie"}[ba]
        flips += ab_winner != ba_winner
    return {
        "position_bias": abs(first_wins / decisive - 0.5),
        "flip_rate": flips / len(items),
    }

The paper defines position bias as the absolute deviation of P(A wins) from 0.5 over paired AB+BA evaluations without specifying tie handling; we exclude ties from the denominator so the statistic stays centred on 0.5 when the judge is unbiased. Flip rate is the complementary item-level view — the fraction of pairs where swapping the order changes the answer. Expect it to get worse on harder items: the cohort median flip rate rose from 0.09 on MT-Bench to 0.17 on JudgeBench.

What to change on Monday

The paper distils its findings into a Minimum Viable Validation Protocol, and it is short enough to adopt in an afternoon. Report Cohen’s κ or Krippendorff’s α alongside any exact-match figure and treat the chance-corrected number as the headline. Measure position bias with paired AB+BA evaluations. Replicate over at least three independent runs at temperature 0 with response caching disabled, or your replicates are measuring your cache. Cross-validate on at least two benchmarks that span preference-style and correctness-style label distributions. And when test-retest exceeds 0.95, verify position bias is below 0.10 before you claim reliability, because high stability with high bias is a failure mode rather than a strength.

Computing the headline number is three lines and an import:

from sklearn.metrics import cohen_kappa_score

# One entry per item, drawn from the same label space.
exact_match = sum(h == j for h, j in zip(human, judge)) / len(human)
kappa = cohen_kappa_score(human, judge)
print(f"EM {exact_match:.3f}  kappa {kappa:.3f}  gap {100 * (exact_match - kappa):.1f}pp")

Two things we would add from our own side. Report the label distribution of your eval set next to the κ, because without it nobody reading the number can tell whether a low κ means a weak judge or a skewed dataset. And do not delete the exact-match figure — keep both, with the gap between them shown explicitly. The gap is a useful diagnostic in its own right, and a metric that quietly disappears from a dashboard tends to reappear in a slide six months later with no methodology attached.

One caveat on where to spend effort. Verbosity bias, the classic worry that judges reward longer answers, came out small across this entire cohort: below 0.011 for all 21 judges, with the largest at 0.010. The authors are careful that this was measured under a single pairwise rubric and one operationalisation of length, so it is not a claim that the problem is solved. But on the evidence available, position bias is the one to instrument first.

What we’d do differently

We would have computed κ at the moment we built the hallucination pipeline’s evaluation set, which is when it costs nothing — the labels and the predictions are both already in front of you, and it is a dozen lines. The version of this post we would rather be writing reports our actual κ and how far below 0.92 it sat. This one cannot, and that is a self-inflicted gap.

We would also have separated two things we had merged. Our eval harness treated “the judge is stable” and “the judge is right” as one property, checked with one rerun, because stability is trivially easy to measure and correctness is not. That is precisely the substitution the consistency–bias paradox punishes. The position swap costs one extra API call per item and would have caught it.

The one we are least comfortable with: we compared candidate judges on a single internal preference-style eval set and picked a winner on a margin of a few percentage points of raw agreement. Given a compressed benchmark and an uncorrected metric, we now think that comparison was measuring noise, and the honest description of how we chose that judge is that we did not really choose it.

References

We build and validate LLM evaluation pipelines as part of our AI & Machine Learning Engineering practice. Talk to us if you have a judge in production whose validation you cannot fully defend.