Projects · Evaluation integrity · Research prototype · MEASURED

inspect-audit

What this establishes

That a read-only auditor can flag silent validity failures in Inspect evaluation logs, samples dropped from a metric's denominator, grader parse-failures scored as data, a model grading its own output, duplicates, truncation, and non-reproducible settings.

What this does not establish

A PASS is not a validity certificate. A biased dataset, a mis-specified task, a subtly-wrong-but-parseable grade, or contamination are out of scope. NOT_CHECKED is not PASS; there are no model calls and no language-model judge.

Trusted base

The author's check catalog and its thresholds, which are an authored coding scheme and part of the trusted base; the inspect_ai schema; and heuristic signals labelled as such.

Method

24 static checks run over the log, each firing only on evidence, with catalog-to-code consistency test-enforced; the verdict per check is FAIL, WARN, PASS or NOT_CHECKED.

Result

58 tests pass; on the broken fixture, 3 FAIL and 1 WARN over 23 checks (exit 2); on the clean fixture, 0 FAIL (exit 0).

Reproduce

pip install -e '.[dev]' && inspect-audit examples/broken.eval

Toolchain: Python 3.11+, inspect_ai; static, no model calls, never modifies the log
Verified at: repowazdogz-droid/inspect-audit@538480d4 (2026-07-20)

This pass. Re-verified this pass: the audit and the test suite were re-run at 538480d4 (broken to FAIL, clean to PASS, 58 tests passing).

Independent reproduction

None known.

Prior art and positioning

Static analysis of evaluation logs and validity threats in measurement. The contribution is the catalog of silent-failure checks with NOT_CHECKED kept distinct from PASS.