Two graders agreed, and both were wrong

The convenient way to check whether a model got an answer right is to ask another model. Cheaper still, ask two, and take their agreement as reassurance. If both judges say the answer passed, surely it passed. That inference is where the trouble lives, because agreement between two graders is a fact about the graders, not about the answer.

Run real grade-school maths answers past a pair of language-model judges and the seams show. The two do not fail at random. One is systematically too strict: it fails answers that were in fact correct. The other is systematically too lenient: it passes answers that were in fact wrong. Averaged into a single accuracy figure they look competent and roughly interchangeable. Read apart, they are biased in opposite directions, and a single number hides that the same score can be produced by a grader that is too harsh and one that is too soft. The figure below shows the two against the truth, erring on opposite sides.

The sharper lesson comes from the case where two graders agree and are both wrong together. A language-model judge passes an answer; a plain string-matching grader passes it too; the answer is wrong. Two independent-looking checks concur, and their concurrence is worth nothing, because both were fooled by the same surface feature. The agreement is real. It is simply not evidence of correctness. It is evidence that the two graders share a blind spot.

There is a second twist worth naming. One of the judges also generated some of the answers it was later asked to grade. A grader marking its own work is not an independent check; part of what its verdict measures is its own preference for its own output. The evaluation does not correct for this, and says so.

None of this is a claim that the judges are useless, or that the numbers generalise. The corpus is small, the ground-truth labels were adjudicated by an assistant rather than by people, and the effect is reported as measured on that corpus, not as a law. What it does show, concretely, is the inference failure the programme calls convergence mistaken for correctness: independent methods agree, and the agreement is read as truth. The remedy is unglamorous, score each grader against known answers on its own before comparing them to each other, and it is the kind of check that would have caught this.

This case is the evaluation-integrity corner of a programme about whether claims about AI systems can be independently checked. Read where it fits in the programme.