Projects · Evaluation integrity · Research prototype · MEASURED

inspect-replay

What this establishes

That two Inspect AI evaluation logs can be compared deterministically and sample-aligned, reporting what changed together while distinguishing 'unchanged' from 'cannot tell'.

What this does not establish

No causation, co-occurrence only. It does not re-run models; a numeric score change is not labelled a regression; different scorers are not comparable.

Trusted base

The inspect_ai log parser and schema; the author's alignment-key logic, flagged in-repo as the riskiest module; and the UNKNOWN, NOT_CHECKED and NOT_COMPARABLE ignorance model.

Method

It reads two recorded evaluation logs, aligns samples, and reports per-sample changes with an explicit ignorance state; the output is deterministic and byte-identical, read-only, and enforced by tests.

Result

117 tests pass; the example diff reports CHANGED with the honesty line that no recorded config field accounts for the change.

Reproduce

pip install -e '.[dev]' && inspect-replay compare examples/baseline.eval examples/sample-regression.eval

Toolchain: Python 3.11+, inspect_ai >=0.3.180; offline, read-only
Verified at: repowazdogz-droid/inspect-replay@8be423cd (2026-07-20)

This pass. Re-verified this pass: the compare command and the test suite were re-run at 8be423cd (byte-identical output, 117 tests passing).

Independent reproduction

None known.

Prior art and positioning

Structured diffing and evaluation reproducibility. The contribution is the deliberate 'unchanged versus cannot tell' distinction with a build-time guard against overclaiming causation.