Research / Inspect AI
The same failing run reported 6 of 6 or 4 of 6 completed samples, depending only on scorer order.
ReproducedFixed upstream- Result
- The same failing run reported 6 of 6 or 4 of 6 completed samples depending only on scorer order.
- Boundary
- One framework behaviour, one failing run, a mock model. Says nothing about any model score. The fix was written by another contributor.
- Who did what
- Warren reported the behaviour and supplied the reproducer (issue #4602). Another contributor wrote and submitted the fix (PR #4604); OMEGA did not implement it.
- Public evidence
- Issue #4602Merged fix #4604 (another contributor)
Interactive figure. Every number it draws is stated in the record below.
The question
Does a finished evaluation report the same number of completed samples whatever order its scorers are listed in?
Inspect AI is a widely used framework for evaluating language models. completed_samples is a default column when many logs are analysed together, so two runs with identical failures would report different completion totals and carry the difference into analysis.
Reproduced: A reported behaviour was reproduced from a minimal, public input. Fixed upstream: The maintainers accepted a fix into the project. Who wrote it is stated on each case.
What was tested
- Built a six-sample run with two scorers, the second raising an error on two samples, using a mock model.
- Declared the same scorers in both orders, changing nothing else.
- Compared results.completed_samples with the errors recorded in the sample log.
What happened
- [cheap_scorer, flaky_grader] reported 6 of 6 completed. [flaky_grader, cheap_scorer] reported 4 of 6. Both runs ended with status=success, and the same two samples carried an error in each.
- Scoring stops at the exception, and since 0.3.247 a sample keeps any scores recorded before it. The field was therefore counting samples with at least one retained score, so its value depended on whether a scorer ran before the one that raised.
- Releases 0.3.245 and 0.3.246 reported 4 for both orderings; the difference first shipped in 0.3.247.
Challenge
The objections that would most easily undo the conclusion, and what answered each.
- Was anything else different between the two runs?
- No. Same dataset, samples, scorer implementations, model and failures. Only the order of the scorer list changed.
- Is it a one-off?
- The count followed the same rule across four scorer arrangements and several releases, and retries did not change it.
What survived
Did not survive
- completed_samples is independent of scorer order.6 of 6 under one order, 4 of 6 under the other, for identical failures. Fixed upstream in pull request #4604, by another contributor.
Survived
No claim was confirmed. The investigation established a defect, not a guarantee.
Not established, or not tested
- Nothing here concerns the correctness of any model score, how often the condition arises in practice, or other evaluation frameworks.
- The upstream decision to score errored samples is separate and was not disputed.
Limits
One framework behaviour, one failing run, mock model.
Reproduce or inspect
- Source
- UKGovernmentBEIS/inspect_ai/issues/4602
- Revision
inspect_ai 0.3.249 (release tested)- Command
pip install inspect_ai && inspect eval repro.py --model mockllm/model- Expect
completed_samples 6 of 6 or 4 of 6 depending on scorer order (before the fix)
repro.py is given in full in issue #4602. The command is given as the repository documents it; it was not re-run for this page.