Method

Two views over one practice. The loop is how a claim is turned into an instrument, a control and a graded result. The spine is where, in the life of a governed action, such a claim can be checked at all.

1. 1

   Claim

   Start from a sentence someone relies on. A test suite is green. A monitor prevents. A judge passes. A proof covers the design. The claim is written down as stated, not as it would need to be to be checkable.

   For example. “The system cannot write more than 4 KB.” “The dashboard shows 1,026 preventions.” “Every agent stayed inside its allowance.”

2. 2

   Subject

   Name what the claim is about and what kind of object it is: a program built for the study, a copy of a real artifact, a third-party library or framework, or a formal model. The grade a result can reach depends on this, so it is recorded on every finding.

   For example. An authored MCP server. A sandbox copy of a real scheduled-job file. Inspect AI itself. A Lean model of an escrow protocol.

3. 3

   Instrument

   Build the thing that can see the failure if it is there: a second writer, an independent read-back, a solver, a mutation harness, a ledger the executor does not control. The instrument is chosen for what it observes, not for what it reports.

   For example. An executed-action log written by a second process. A Z3 encoding of the policy. Two hundred netlist mutations.

4. 4

   Negative control

   Before a pass is allowed to mean anything, feed the instrument a case known to be bad and confirm it goes red. A check that has only ever been seen to pass has not been tested. Where no control exists the finding says NONE or NOT APPLICABLE rather than inventing one.

   For example. A forced-breach cell that must read 1.000. Four Lean theorems that must fail when a guard is removed. A seven-class leak injected into this site's own build.

5. 5

   Intervention

   Change one thing the claim depends on and hold the rest fixed: move the control from beside the path to on it, hand the executor a permit that was never issued, remove the line the prompt was assumed not to need.

   For example. Two arms differing only in enforcement position. Three forgeries of increasing strength. One redundant sentence appended to a prompt.

6. 6

   Observation

   Measure the effect from outside the thing that produced it, and record the expectation before the measurement wherever that is possible. Where an expectation had to be written after the fact, the record says so.

   For example. Bytes hashed before and after. The environment's own ledger, never the model's text. A verdict row that carries expectation_set: after_measurement.

7. 7

   Verdict

   Grade the result by how far the evidence reaches: PROVEN inside a named frame, MEASURED with a stated sample, OBSERVED in one instance, or OPEN. The grade fixes the verb a result may use.

   For example. A kernel-checked bound is PROVEN over its model. 60 of 60 episodes is MEASURED for that model on that day. One forged permit accepted is OBSERVED.

8. 8

   Limitation

   State what the result does not establish, at the same weight as the claim. This is the line most readers actually need, and it is never in a footnote.

   For example. Not a measurement of deployed systems. Terminal-observation safety, not every intermediate state. One phrasing, one day, three models.

9. 9

   Reproduction

   Publish the artifact and the command that re-derives the number, or say plainly that the artifact is not public and what that costs. Independent reproduction by someone else is recorded separately and is currently none for every result here.

   For example. make check. python3 reproduce.py. A bundle of records and pointers where the artifact cannot be published.

Five steps. Each one exists because skipping it is a way that real systems have already gone wrong.

1. 1

   Authorise

   Decide in advance exactly what the system is allowed to do, and write that decision down before it acts.

   Why it is a step. If permission is worked out at the moment of action, or inferred afterwards from what happened, there is nothing independent to check the action against.

   3 supporting results: PROVEN A spending cap was proved to hold in every state the system can reach, not just the ones anyone tested. · PROVEN A limit on what an agent may do was proved to hold no matter how concurrent operations interleave. · OBSERVED An agent was limited to 4 KB. It wrote 100 KB. Every test passed, an independent replay verifier said VERIFIED, and a checked invariant held.

2. 2

   Execute

   The system acts, and the action it takes is the one that was authorised, not merely one that resembles it.

   Why it is a step. The gap between the action a policy approved and the action that actually ran is where several of the failures below live.

   2 supporting results: OBSERVED A system passed all 24 of its recorded tests and still had a serious flaw. · OBSERVED Tools removed from the list an agent is shown were still there to call by name.

3. 3

   Observe

   Measure what actually changed, by looking at the files, records or state afterwards.

   Why it is a step. A record of what a system intended reads exactly like a record of what it did, until something measures the difference.

   1 supporting result: MEASURED A dashboard reported 1,026 actions prevented. All 1,026 had already executed.

4. 4

   Verify

   Check the observation against the permission, mechanically, using something that was not part of doing the work.

   Why it is a step. Checkers can be wrong, and two checkers that share a source are wrong together.

   4 supporting results: MEASURED Two off-the-shelf AI graders scoring the same work were wrong in opposite directions. · MEASURED Two AI graders agreeing tells you less than it appears to, because part of that agreement is forced by both being right. · OBSERVED The same failing evaluation reported a different number of finished samples when its scoring steps were listed in a different order. · OBSERVED A widely used library reads a number, writes it back out, and the two are not the same.

5. 5

   Replay

   Someone else can re-run the check later, from the record alone, and get the same answer.

   Why it is a step. Evidence that only its author can reproduce is testimony. Evidence anyone can re-derive is a different kind of thing.

   2 supporting results: PROVEN Changing a sealed record after the fact is provably as hard as breaking the hash it is sealed with. · MEASURED A training run re-derives its exact result from its recorded inputs on one machine. Across different hardware it stays unknown.

The five stages describe where a governed action can be checked: what was authorised, what ran, what changed, whether the two agree, and whether anyone else can re-derive that agreement later. The loop above describes how each of those checks is itself tested. Every finding on this site sits at one stage of the spine and was produced by one pass of the loop, so a stage on the method page links to the findings that support it and each finding links back to its mechanism on the how-checks-fail page.

Where a study could be committed before its data existed, it was: the design, the sample size, the controls and their pass conditions, the hypotheses and what would falsify each. Later conditions were added as dated amendments registered before their data. Every departure is logged with its timing. Where no preregistration exists, the finding says so rather than implying one.

The site is built by a generator that derives every count from the records; a count cannot be typed. The build halts if a coverage bar does not partition, if a finding has no limitation, if an empirical grade has no denominator, or if a repository reference is malformed. A publication audit runs in series after the build and fails on home paths, hostnames, lab node names, credentials and any external resource. Each of those rules is exercised by a negative control that injects the leak alone and requires the audit to catch it, and the build gates are exercised by mutations that must make the build fail.

25 leak classes injected one at a time; the build-gate mutations are listed in tools/gate-controls.mjs. The site's own counts: 18 findings, 19 artifacts, 15 findings with a present negative control, all derived at build time.

How claims are gradedHow checks fail