Findings · Evaluation integrity
In a widely used evaluation framework, the same failing run reported a different number of completed samples depending only on the order its scorers were declared in: 6 of 6 under one ordering and 4 of 6 under the other, with both runs ending in a success status.
Evidence
A finished evaluation run records how many of its samples completed. That number is what a reader consults when deciding whether a result covers the dataset it claims to cover. Here the number moved when nothing about the work moved: same dataset, same model, same scorer failures, only the declaration order changed.
The condition surfaced while building a read-only auditor for evaluation logs, which is the ordinary way this kind of defect appears. Nothing was wrong with the model, and nothing was wrong with the scores. The bookkeeping a run keeps about itself disagreed with the run.
It was reported upstream on 23 July 2026 and fixed by the maintainers on 29 July 2026. The fix is theirs, not this programme’s; what is recorded here is the report and the condition it described.
Sample. Two tasks over one 6-sample dataset, differing only in scorer declaration order, with the second scorer raising on samples 2 and 5. A constructed reproducer, not a survey.
Boundary
One defect in one framework, in the metadata a finished run records about itself. It says nothing about the correctness of any model score, nothing about how often the condition arises in practice, and nothing about other evaluation frameworks. The upstream design decision to score errored samples is separate and was not disputed.
Independent reproduction. Reproduced by the framework maintainers, who merged a fix as pull request #4604 on 29 July 2026, six days after the report.
Reproduction
pip install inspect_ai && inspect eval repro.py --model mockllm/model- Toolchain
- Python, inspect_ai at the pre-fix commit, the mockllm provider; no API key and no network
- Expected output
- results.completed_samples reports 6 with the passing scorer declared first and 4 with the raising scorer declared first; status is success in both runs and samples 2 and 5 carry an error
- Claim stated at
- UKGovernmentBEIS/inspect_ai · github.com/UKGovernmentBEIS/inspect_ai/issues/4602; fix in src/inspect_ai/_eval/task/results.py
- Verified at
- UKGovernmentBEIS/inspect_ai@f0e57a7c (2026-07-23)
This pass. Confirmed upstream rather than re-run in this pass: the issue was filed 23 July 2026 and closed by the merged fix on 29 July 2026. The behaviour described here is the behaviour at the pre-fix commit.
Artifact: UKGovernmentBEIS/inspect_ai