Agreement and accuracy in language-model judging

Two pre-registered experiments on language-model-judge reliability, run over 600 items with machine-checkable ground truth (GSM8K integer keys and MBPP unit-test execution) and 19 judges. No judge grades an answer it generated, and that exclusion is enforced in code rather than by convention.

What it establishes

Across 171 judge pairs, judge errors are correlated: mean phi 0.567, with a judge-clustered bootstrap interval of [0.483, 0.638], and every pair positive. A panel of these judges is therefore worth less than its headcount, because the judges fail on the same items. On this roster high agreement did not admit low accuracy, so what agreement overstates here is independence rather than accuracy.

A second experiment measures stacked oversight. When a meta-judge grades another judge’s verdict without seeing ground truth, sharing a vendor with that judge makes it worse: it catches fewer of that judge’s false passes, a paired difference of −0.102, a meta-judge-clustered bootstrap interval of [−0.151, −0.051], and a sign-flip p of 0.011. Self-grading is the extreme case.

One result is analytic rather than empirical, and is stated separately because it constrains how the others should be read. For binary verdicts against binary truth, two judges that are both wrong on an item must agree on it, so P_agree ≥ accA + accB − 1. Agreement is not free to vary independently of accuracy, and a raw correlation between agreement and accuracy is partly forced by that floor. What remains free is the excess over independence, 2·Cov(a,b), which is the error correlation.

What it does not establish

It is not a capability benchmark. The numbers are facts about 600 items and these judge checkpoints. Ground truth is a machine key rather than human labelling, so the property judged is final-answer correctness, not reasoning validity. Whether the meta-judge effect is agreeableness or selective blindness is undetermined by this data, and is reported as inconclusive rather than as a finding. “Shares a base model” is a checkable claim only for the open-weight judges and a declared-vendor proxy otherwise.

Reproduction

Every number recomputes from committed raw model outputs with no model access. The pre-registration and both amendments were each committed before the data they govern existed, every departure is logged with its timing, and a mechanical sweep checks each figure in the write-up against the field of the report that produces it, including a residual scan so that an unmapped number cannot pass silently.

Full method, boundaries and figures: evaltrust at v0.2, document writeup/WRITEUP-AGREEMENT.md.