evaltrust

That two off-the-shelf language-model judges disagree with the ground truth in opposite directions on a small GSM8K corpus, and that at that sample size a zero-disagreement judge's agreement interval is degenerate, measured, with digests that recompute offline.

Judge reliability. The per-answer correctness labels are assistant-adjudicated, not human; two of the three judges also generated answers, so self-preference is uncorrected; the sample is small.

Five graders, three language-model judges, a substring matcher, a numeric oracle, and an ensemble, score 45 real GSM8K trajectories, and every number recomputes from committed data under a digest check.

gemma3 disagrees on 4 of 45, all too strict; llama3 on 2 of 45, all too lenient; the ensemble's agreement interval is degenerate at zero disagreements; the report digest is 62bb20c4.

The objective GSM8K integer answer key; the assistant-adjudicated per-trajectory labels, which are an authored coding scheme and part of the trusted base; the local Ollama judge models at temperature 0.

Language-model-as-judge evaluation and inter-rater agreement. The contribution is the opposite-direction failure and the degenerate-interval honesty, with every number recomputing from committed data.

python scripts/build_report.py && python scripts/recompute.py && pytest -q

Findings drawn from this artifact are on the evidence ledger.