Projects · Evaluation integrity · Research prototype · MEASURED

evaltrust

What this establishes

That two off-the-shelf language-model judges disagree with the ground truth in opposite directions on a small GSM8K corpus, and that at that sample size a zero-disagreement judge's agreement interval is degenerate, measured, with digests that recompute offline.

What this does not establish

Judge reliability. The per-answer correctness labels are assistant-adjudicated, not human; two of the three judges also generated answers, so self-preference is uncorrected; the sample is small.

Trusted base

The objective GSM8K integer answer key; the assistant-adjudicated per-trajectory labels, which are an authored coding scheme and part of the trusted base; the local Ollama judge models at temperature 0.

Method

Five graders, three language-model judges, a substring matcher, a numeric oracle, and an ensemble, score 45 real GSM8K trajectories, and every number recomputes from committed data under a digest check.

Result

gemma3 disagrees on 4 of 45, all too strict; llama3 on 2 of 45, all too lenient; the ensemble's agreement interval is degenerate at zero disagreements; the report digest is 62bb20c4.

Reproduce

python scripts/build_report.py && python scripts/recompute.py && pytest -q

Toolchain: Python >=3.11, numpy, scipy; a local Ollama is needed only to regenerate verdicts, not to reproduce the committed numbers
Verified at: repowazdogz-droid/evaltrust@18e999a5 (2026-07-20)

This pass. Confirmed by source read at 18e999a5 (the committed digest and report were verified present); the recompute pipeline was not re-run this pass.

Independent reproduction

None known.

Prior art and positioning

Language-model-as-judge evaluation and inter-rater agreement. The contribution is the opposite-direction failure and the degenerate-interval honesty, with every number recomputing from committed data.