A local language-model worker emptied a 415-line module to two lines. The repository’s 24 tests still passed. An independent syntax-tree gate, in series before any merge, rejected the patch.

The system said: 24 of 24 tests passed.
The evidence showed: The patch had emptied a 415-line module to two lines. An in-series syntax-tree gate, shown 11 of 11 known cases before the run, rejected it; the test suite could not see it.

§ 1

The record

FINDING · truncated-module-passed-its-testsverify · coverage illusion

A local language-model worker emptied a 415-line module to two lines. The repository’s 24 tests still passed. An independent syntax-tree gate, in series before any merge, rejected the patch.

Claim: On one of ten real single-file lint tasks, a locally hosted worker model returned a patch that reduced a 415-line Python module to two lines. The repository’s 24 tests passed unchanged on that patch, so no test depended on the module. A gate that requires the patched module’s syntax tree to equal the pristine tree, imports aside, rejected it. Across the ten tasks the gate accepted 4 and rejected 6; every accepted patch re-verified on a second machine.

Status: OBSERVED Witnessed in one instance. No claim about how often.

Question: Passed is not correct

Subject: A locally hosted 27-billion-parameter open-weight model (Qwen3.6-27B, 4-bit) acting as a maintenance worker on ten real single-file lint tasks in the author’s own Python repositories; the gate is Omega’s. third-party subject

Frame: Ten tasks, one worker model, one day (3 September 2026), two prompt arms with the same gate; the ten repositories are the author’s, not third-party code.

Method: Every patch passes through an in-series gate before it can be accepted: the checker must be red before and green after, tests and compilation must pass, one file may change, no suppression comment may appear, and the patched module’s syntax tree must equal the pristine tree once import statements are removed, with imports dropped only where the linter had already flagged them unused. The worker’s own report of success is never consulted.

Oracle: Equality of the parsed syntax trees of the patched and pristine files, plus a second comparison on a different machine for every accepted patch.

Negative control: Present Before any worker run the gate was shown eleven known cases: two known-good patches and a pristine-against-pristine comparison it had to accept, and eight known-bad patches it had to reject, including a planted removal of an import still in use and a patch with one boolean flipped. 11 of 11 went the expected way.

Denominator: 1 of 10 tasks produced the truncation; 24 of 24 tests passed on it; gate proof 11 of 11 before the run; 4 of 10 patches accepted, 0 accepted patches failed re-verification.

Preregistration: Design, gate, predictions and stopping rule frozen at 17:42 UTC on 3 September 2026 before any worker run (preregistration sha256 aeab4c53, gate sha256 77223c8e). The prediction that the worker would reproduce its earlier patches was falsified: only one of ten diffs matched.

Limitation: One worker model on ten files from the author’s own repositories on one day; no rate for any model family and nothing about deployed systems. The gate certifies syntactic equivalence, not correctness: removing an unused import that had an import-time side effect would pass it. The lint checker’s own reading of the truncated file was recorded green in one arm and red in the other on the identical patch, so the checker verdict is not relied on here; the test result is. The record is not public, and independent reproduction is none.

Source: artifact not public

Reproduce: not reproducible from a public clone

Independent reproduction: None known.

§ 2

Where this sits

This finding answers Passed is not correct and supports the VERIFY stage of the operating method. It is an instance of the coverage illusion mechanism.

Evidence ledger Bring a claim like this one