OBSERVED
Finding · verify
The same failing evaluation reported a different number of finished samples when its scoring steps were listed in a different order.
§ 1
The record
FINDING · completed-sample-count-depends-on-scorer-orderverify · self certification
The same failing evaluation reported a different number of finished samples when its scoring steps were listed in a different order.
- Claim
- In a widely used evaluation framework, the same failing run reported a different number of completed samples depending only on the order its scorers were declared in: 6 of 6 under one ordering and 4 of 6 under the other, with both runs ending in a success status.
- Status
- OBSERVED Witnessed in one instance. No claim about how often.
- Question
- Passed is not correct
- Subject
- Inspect AI, a widely used evaluation framework (third-party system). third-party subject
- Frame
- One failing run, two scorer orderings, mock model.
- Method
- Same run declared with scorers in two orders; the completed-sample metadata compared.
- Oracle
- The run metadata the framework writes about itself, compared across orderings.
- Negative control
- Not applicable The two orderings are each other's control; nothing else is varied.
- Denominator
- 6 of 6 under one ordering, 4 of 6 under the other, both with a success status.
- Limitation
- One defect in one framework, in the metadata a finished run records about itself. It says nothing about the correctness of any model score, nothing about how often the condition arises in practice, and nothing about other evaluation frameworks. The upstream design decision to score errored samples is separate and was not disputed.
- Source
- UKGovernmentBEIS/inspect_ai @ f0e57a7c
- Reproduce
pip install inspect_ai && inspect eval repro.py --model mockllm/model
- Independent reproduction
- None known.
- Upstream
- MERGED UPSTREAM Fixed upstream in inspect_ai pull request #4604. Link
§ 2
Where this sits
This finding answers Passed is not correct and supports the VERIFY stage of the operating method. It is an instance of the self certification mechanism.