Count the evidence.
46 verified public results. 40 with public checking instructions. 12 accepted fixes and contributions.
Verified minimum across public Omega repositories and technical reports, not an exhaustive count of all work. One row per separately stated, evidenced conclusion. The older site ledger remains an inspectable subset.
Checked 7 October 2026. 4 of the accepted contributions were authored by Omega. The rest were fixes by upstream maintainers or contributors responding to Omega reports.
Counting rules
- Published results
- Count a named scoped conclusion with public result evidence. Count multiple conclusions within one study only when separately stated and evidenced. Deduplicate report/fix pairs and overlapping project/site entries. Exclude private-only results, uncompleted final challenge outcomes, demonstrations without a research conclusion, documentation-only reports, forks without a distinct evidenced contribution, tests, helper lemmas and repeat runs.
- Public checking instructions
- Subset with publicly accessible code/artifacts and explicit instructions to check or reanalyse the stated result. Frozen-data analysis counts; generating identical new provider outputs does not. Toolchains, API access or model downloads may be required. Availability is not a successful fresh reproduction or independent reproduction.
- Accepted contributions
- One distinct reported defect fixed or Omega-authored contribution merged by the external project. Require merged_at plus merge commit, or a directly committed maintainer fix explicitly addressing the report. Exclude open, closed-unmerged and self-merged changes. Acceptance is not endorsement.
No fresh external reproduction is implied. Public provider calls can change; checking frozen records is different from generating identical new outputs.
Every counted result
46 rows; 40 instruction sources. Commit-linked files stay fixed even when a repository changes. GitHub issue bodies can change: their audit timestamp and SHA-256 are recorded in the downloadable inventory.
1. A start permit assembled entirely from public registry facts, with its reference number derived by the authorizer's own function and committed to no ledger, was accepted by the bounded executor and the registry-named transform was applied; the target's bytes changed.Inspect ↓
Source commit: efba311b08bb75f054dd49bf8a4f9e1b6f13d614
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
2. A write of 100,000 bytes executed under a policy that capped writes at 4,096 bytes, while 66 of 66 tests passed, an independent replay verifier returned VERIFIED, and a checked complete-mediation invariant held.Inspect ↓
Source commit: 2ea137b9cf83e509234bc24ab96e70d1f8808f7e
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
3. Total spend never exceeds the cap in any reachable state of the escrow protocol, proved in Lean for any finite set of replicas and any non-negative amounts.Inspect ↓
Source commit: 6595f1283a43b319d62fcb065531a3b303794932
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
4. A widely used C JSON library does not preserve numbers across a serialise-and-reparse round trip: its number pipeline is lossy by design, printing with limited precision and comparing re-reads by tolerance rather than equality.Inspect ↓
Source commit: 92ae42dcc929d0061d75ea863e9b043c55fa500c
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
5. completed_samples was computed from scored samples, so the same failing run reported 6 of 6 or 4 of 6 depending only on scorer declaration order.Inspect ↓
Report snapshot: 2026-07-29T17:12:58Z · SHA-256 d5a82f25e9a75ede3bd180cd9d38a419c63699c5506ab2d48f844bbb7e82c7b6
Public reproduction code/commands in the report; not independent reproduction.
6. A capability meter never exceeds its cap over every concurrent interleaving of charge operations, proved in Iris concurrent separation logic for any cap and any finite list of operations.Inspect ↓
Source commit: dcc8ecbd7b51ad63bcedb2466c6a4187802c6213
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
7. Across 600 items and 171 judge pairs, judge errors correlated positively in 171 of 171 pairs at mean phi 0.567, and the correlation between pairwise agreement and accuracy fell from 0.824 to 0.077 once the algebraically forced component of agreement was removed.Inspect ↓
Source commit: 0803d6b24c83df8372112894cb3508f2c7c83970
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
8. At 45 answers, a judge with zero disagreements has an agreement interval spanning its whole range; the perfect score reads as 'no counterexample found at this sample size', not as demonstrated reliability.Inspect ↓
Source commit: 0803d6b24c83df8372112894cb3508f2c7c83970
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
9. Two language-model judges scoring the same 45 GSM8K answers disagreed with the ground truth in opposite directions: one systematically too strict, the other too lenient.Inspect ↓
Source commit: 0803d6b24c83df8372112894cb3508f2c7c83970
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
10. Over 10,000 injected events, a monitoring arm recorded 1,026 actions as prevented and all 1,026 executed; an in-series arm consuming the same events, evaluator and policy recorded 6 of 9,880.Inspect ↓
Source commit: 08e52549fff10035d2a8c616ef32af5a5ac3bb64
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
11. Tampering with a sealed, hash-linked decision record forces a hash collision: the canonical encoding is injective, the chain is append-only, and detection follows in Lean without a collision-resistance axiom.Inspect ↓
Source commit: 2d1ecfff081c86b50b5bc2ec61e246e9a3eeddd5
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
13. A training run's final loss and trajectory re-derive bit-for-bit from its recorded inputs on one machine; cross-hardware re-derivation is left as UNKNOWN, and not claimed.Inspect ↓
Source commit: b291ee06059b96a9dac1fe953c8326ba301680bf
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
14. In a shared-budget environment where each agent's allowance is individually satisfiable but the allowances sum to 1.5 times the collective budget, three language models produced a compliant collective breach in 60 of 60 BLIND episodes each (two population sizes, 30 episodes per cell), with zero individual allowance violations; one appended sentence restating the same headroom took the rate to 4 of 60, 2 of 60 and 0 of 60.Inspect ↓
Source commit: 876d3368e662aae7fea3480e58385e69804e32eb
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
15. In the complete rerun, six single-decision cases violated an encoded governance property. The language-model judge passed 4 of the 6 across three seeds each; the formal checker proved the violation in 6 of 6, and both verdicts are sealed in the same record.Inspect ↓
Source commit: f22def1e58012ee1883c3643365138873a2117b9
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
16. Five defects were injected into a small power-control design whose formal verification argument was complete and green. Two of them passed every property derived from the original specification: one because the specification required voltage to be sufficient and never minimal, one because a two-step transition is internally consistent. A third escaped every property about the design's own variables while recording something false about the physical world.Inspect ↓
Source commit: e49a0c5e57737d5717bf4307822f788a8359390d
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
17. A Kani harness left its second assertion UNREACHABLE after the first injected bug, so its pass carried no information; a loom test reported 1 passed on a build containing a lost update because the atomic was never loom-instrumented, with byte-identical output for the clean and broken builds; and two Lean models of the same Rust function, one pinning real sizes and one returning 0 for every type, produced byte-identical axiom audits.Inspect ↓
Source commit: 996d248ff873832e8d822d0ad13f8e0f7f9372c5
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
18. In a closed-data Lean 4 model (cap 10, budget 40, six agents) the per-agent-compliant profile [10,10,10,10,10,10] breaches the budget and the conserved meter accepts no breaching profile while still accepting a within-budget one, with no axioms. In a Z3 linear-integer model of the same instance, eleven queries establish that the sum rule holds for every coordinated profile, does not bound a harm equal to one agent's draw when its threshold (15) is below the pool (allocation harm escapes with witness [0,0,16,0,0,0]; a per-recipient cap closes it), and is blind to any coordination channel in the model. This is a contrast between two chosen harms; an earlier statement that the rule bounds a harm if and only if the harm factors through the aggregate is withdrawn, because it fails in both directions in the same encoding.Inspect ↓
Source commit: 618cb1c3004acc6c3dc27b987112ff3f9c7779d5
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
19. The proof went through, and a cover requirement on the decoder's syndrome output showed that the output was declared and never driven. A one-line assignment fixed it; the proof and a width sweep to 64 bits passed with the fix.Inspect ↓
Report snapshot: 2026-09-03T21:23:23Z · SHA-256 390911a4a391633b68901ae0eb726fdedad05342ab8025032abbf12228f6986f
Public formal harness and commands in the accepted PR.
20. In PyBaMM's BasicDFN with a concentration-dependent transference number, an independently assembled lithium inventory drifted by 2.44% of the initial inventory over the reporter's 4,800-second protocol on v26.4.1, main at 258fdc8 and the head of pull request #5524, byte-identical between main and the pull request; the drift persisted under temporal tightening to 1e-10 and spatial refinement to 80 cells, and matched, instant by instant, the rate predicted from the missing transference-gradient term. The constant-transference control conserved to 7.7e-15 mol.Inspect ↓
Source commit: adaa877eda7f3cac22f116a006e11a513e586323
Public code and instructions; toolchain/dependencies may be required. Availability is not independent reproduction.
21. Four of fourteen unsupported groups returned converged=True with no warning and identical non-physical results; ten raised the intended error; one group documented as supported is rejected by the code.Inspect ↓
Report snapshot: 2026-10-06T16:28:59Z · SHA-256 36f1f91d98831ce931553749a2bb9adf8d48ce57223ea34bf52e36140605fd89
Public reproduction code/commands in the report; not independent reproduction.
22. Every called cross-file symbol was emitted as an axiomatised pure function; dafny verify reported 1 verified, 0 errors on code whose rollDie() returns different values per call.Inspect ↓
Report snapshot: 2026-08-25T06:46:52Z · SHA-256 da0d6f609c42602c5459d24dfb9948e1259365c992fc0a63dc0ed8eca6dcc88d
Public reproduction commands/code in the report or PR; provider dependencies are not independent reproduction.
23. Neither the final log nor the live snapshot recorded which path produced each operator judgement, so verdicts from different paths were indistinguishable on disk.Inspect ↓
Report snapshot: 2026-08-31T00:55:11Z · SHA-256 c5211645d57a72b9160988e7742e0cc454ae44c1ba5d1645b30a18a2d62ee2f1
Public reproduction commands/code in the report or PR; provider dependencies are not independent reproduction.
24. Generator and async-generator methods were routed to the coroutine wrapper, so generation was skipped and the span ended at generator creation rather than at consumption.Inspect ↓
Report snapshot: 2026-08-31T13:35:24Z · SHA-256 3418819a8f004c376bd17f41056b53169cbfa0872c9e538d65cc81c352593e45
Public reproduction commands/code in the report or PR; provider dependencies are not independent reproduction.
25. Every LLM-mode scan through the Anthropic provider failed: the schema carried minimum and maximum bounds the structured-output endpoint rejected.Inspect ↓
Report snapshot: 2026-06-16T21:49:54Z · SHA-256 8940e1382bef894e3d830b1873267106bd25b9422d69049336f0f8c3f50f2517
Public reproduction commands/code in the report or PR; provider dependencies are not independent reproduction.
26. Eight phantom-divergence classes surfaced while building the target; the pull request adds a differential target over the typed expression.Inspect ↓
Report snapshot: 2026-08-24T15:56:02Z · SHA-256 c1b650be7f7a8963e8a16a8688df722de5dbbc80feca8c549afab3be953a19e1
Public inspection evidence; no complete reproduction instructions counted.
Not included in the checking-instruction total.
27. The specification carried the policy and embodiment configuration but nothing about the grader, so two runs graded by different models against different rubrics wrote byte-identical specifications. The pull request adds the grader name and its resolved configuration.Inspect ↓
Report snapshot: 2026-09-22T06:38:21Z · SHA-256 f0be4f1d58d4c3e2bff0b4cc66f811290f1fdeef3a954151417d99cabbbbe8de
Public inspection evidence; no complete reproduction instructions counted.
Not included in the checking-instruction total.
28. The documented configuration behaved correctly: every side effect observed out of process had a causally preceding permit for exactly that operation. Two documented configuration requirements, a bundle on each delegated child agent and the bundle first in the middleware list, produce no signal when violated. This is not a vulnerability report: the framework does not claim complete mediation, and the request is for a signal, not for a change in what is enforced.Inspect ↓
Report snapshot: 2026-09-04T08:32:03Z · SHA-256 fc6f7b7b37aea09bf1899708eec19e51496836c1859d55b55b11741084701907
Public inspection evidence; no complete reproduction instructions counted.
Not included in the checking-instruction total.
29. It multiplies by the full-cell thickness (767 µm in the Xu2019 parameter set) where the electrolyte occupies 67 µm, overweighting the term by a factor of 11.45; a project contributor confirmed it and asked for a separate report.Inspect ↓
Report snapshot: 2026-09-22T13:19:26Z · SHA-256 056a3bec9411e55fd07c6f32580ed92938e09cf218bcd4196c485131a8e38e81
Public reproduction commands/code in the report or PR; provider dependencies are not independent reproduction.
30. cvc5 returned sat for a formula whose spatial constraints require incompatible heaps. An upstream contributor reproduced the result and identified a polarity-handling cause.Inspect ↓
Report snapshot: 2026-09-28T13:17:47Z · SHA-256 6950d3482ff8c2ce9f8028ebc2d40423ff21848b7a1a85145267b963689ecb36
Public reproduction commands/code in the report or PR; provider dependencies are not independent reproduction.
31. The cached polarity could be wrong, producing an incorrect result on the upstream reproducer. The proposed fix changes the cache key and adds regression tests.Inspect ↓
Report snapshot: 2026-10-01T17:30:23Z · SHA-256 e5b22d1bfefebf7185145379f628531a8d6f5c801635920e0a06b0d47fb621e4
Public reproduction commands/code in the report or PR; provider dependencies are not independent reproduction.
32. mpmath's interval exponential excluded a high-precision endpoint value on a public reproducer; independent Decimal and Arb computations agreed on the exclusion. A maintainer confirmed the problem.Inspect ↓
Report snapshot: 2026-10-02T01:23:59Z · SHA-256 dcdac3f812da7307c6b19cb05ba0f0005738056f44a7bbf10fb77a25aed99a47
Public reproduction commands/code in the report or PR; provider dependencies are not independent reproduction.
33. The command could exit successfully without invoking the checker, so automation could accept a non-check as a pass.Inspect ↓
Report snapshot: 2026-09-10T16:01:45Z · SHA-256 7fd4e0c666b73d921444c2dc334fe1034e8d273e7fda7914338c9de21182e295
Public reproduction commands/code in the report or PR; provider dependencies are not independent reproduction.
34. The provider imported the Anthropic SDK, but the package did not declare that dependency, so a clean documented installation failed at import time.Inspect ↓
Report snapshot: 2026-09-17T10:23:48Z · SHA-256 270461cd17da007791d405bab3420dc28c76ab572d641c2fbd62e89c8a911b2a
Public reproduction commands/code in the report or PR; provider dependencies are not independent reproduction.
35. Per-agent guarantees compose into a checked shared-budget and mutual-exclusion guarantee for the specified fixed-roster model.Inspect ↓
36. The typed transfer protocol checks communication structure; it does not establish the arithmetic budget bound.Inspect ↓
37. The study’s text-only refusal detector missed structured refusals in frozen Claude records; a structured rule recovered them.Inspect ↓
Source commit: fbf2aafb9662a0661c22af50411089dc6cab6b12
python3 reproduce_table4.py
38. The demo lexical checker accepted paraphrases that still communicated a judgement in the bundled synthetic study.Inspect ↓
Source commit: bdf0bde0c2d21db51a2f0a3d04535cc80c850ee2
case-notes-analyse examples/sample_logs
39. A matched-base audit found the published model organism’s scoped preference shift and exposed the base-model confound.Inspect ↓
Source commit: 4906406b04e66a0f802aa61fff9ea55438e437f4
python3 confirm_A/analyse.py
40. The scripted-policy experiment matched its predictions on fourteen cells, with eight controls.Inspect ↓
Source commit: 8707ae35e820315bf6a8b1cd73bffddad24770bf
Public result reports and retained world-state artifacts; no complete result-reproduction instructions counted.
Not included in the checking-instruction total.
41. The model experiment failed its across-family criterion; returning action outcomes was necessary and not sufficient in the tested runs.Inspect ↓
Source commit: 8707ae35e820315bf6a8b1cd73bffddad24770bf
Public result reports and retained world-state artifacts; no complete result-reproduction instructions counted.
Not included in the checking-instruction total.
42. For the published small model, the same backdoor preserved clean accuracy but was refused by the in-series integrity gate.Inspect ↓
43. The constructed composition exfiltrated weights despite individual requests being allowed; the in-series flow guard blocked it.Inspect ↓
44. On the study’s out-of-vocabulary generated attacks the rule detector caught zero of forty.Inspect ↓
45. The scripted real-tool harness exceeded its limit with stale reads; the same calls were denied by the live-counter variant.Inspect ↓
Source commit: 123ae5605283012c2798eacb05862d0b838292cb
python test_harness.py
46. The released proof-checker gate records verdict agreement within its declared scored test scope; no soundness claim is counted.Inspect ↓
Source commit: 3420f5a6ed28fa9025ec610ee6ef2db3340d6dce
Frozen public gate records; no full reproduction instructions counted.
Not included in the checking-instruction total.
Every accepted change
Inspect AI (UK AI Security Institute evaluation framework)
Report or contribution · Accepted change
Merged 2026-07-29 · Authored upstreamMerge/fix commit:
77d936b8b04c3cc3ba0a576a8883c6decdda56c5LemmaScript (a verified-language toolchain emitting Dafny and Lean)
Report or contribution · Accepted change
Merged 2026-08-25 · Authored upstreamMerge/fix commit:
830e7733f44c70f81a35796767728518a87a680binspect-robots (robocurve embodied-evaluation harness)
Report or contribution · Accepted change
Merged 2026-08-31 · Authored upstreamMerge/fix commit:
16eddde5dec2037ba69d1196eb17d745357bad54NVIDIA-NeMo labs-OO-Agents
Report or contribution · Accepted change
Merged 2026-08-31 · Authored upstreamMerge/fix commit:
f8b0c0f46aeb88ccce90ae72c0bd095493414238NVIDIA SkillSpector
Report or contribution · Accepted change
Merged 2026-06-16 · Authored upstreamMerge/fix commit:
a5092dd9b9521ff57a9b53612bb129ce78019002pulp-platform common_cells (an open hardware IP library)
Report or contribution · Accepted change
Merged 2026-09-03 · Authored by OmegaMerge/fix commit:
a257b714450bff120f52a85cf912dc778a5bb988inspect-robots (robocurve embodied-evaluation harness)
Report or contribution · Accepted change
Merged 2026-09-21 · Authored by OmegaMerge/fix commit:
79c782c3e380f462a4369ff7cf2c876416456a34PyBaMM (an open-source battery modelling library)
Report or contribution · Accepted change
Merged 2026-09-18 · Authored by OmegaMerge/fix commit:
6ceca11577231d3cb26f3f58ed80f6b04b6f2554PyBaMM (an open-source battery modelling library)
Report or contribution · Accepted change
Merged 2026-09-22 · Authored upstreamMerge/fix commit:
9daad126258db8f6e621e4746cd3bdac062e7f1fcvc5 (an SMT solver)
Report or contribution · Accepted change
Merged 2026-10-01 · Authored by OmegaMerge/fix commit:
7467ffe6f725e7e83c3f67b4edd407fcb71b1367con-leche (a Lean proof-checking tool)
Report or contribution · Accepted change
Direct maintainer fix · Authored upstreamMerge/fix commit:
19a2f243d90c07f9c3cbded74d20e89727e80d23Orbit (an agent runtime)
Report or contribution · Accepted change
Merged 2026-09-17 · Authored upstreamMerge/fix commit:
43c64817b969ba0273d698aecaee70741d2b461b
Another inspectable number
40 theorem declarations in the shared-capability proof study have recorded closed assumption audits.
Forty declarations with recorded closed assumption audits in one study. Not forty investigations; no fresh kernel execution or independent reproduction in this website audit.
What this leaves out
Four existing site-ledger results have no public artifact verified in this audit. Their original records remain available, with their limitations.
- A local language-model worker emptied a 415-line module to two lines. The repository’s 24 tests still passed. An independent syntax-tree gate, in series before any merge, rejected the patch.
- A proposed analytic gradient for a quantum-chemistry method matched independently computed finite differences in 22 of 22 cells; the released code path failed in 22 of 22. One of the referee's own thresholds fired on both arms alike and was reported as an instrument defect.
- A compiled PyTorch function returned a result computed for a different object state, with no recompilation, where the uncompiled function returned the right one. The investigation was carried out by a locally hosted model with nothing leaving the machine.
- A convincing-looking Unreal validator was rejected because it reconstructed transforms from the source CSV instead of observing instantiated geometry. Three planted geometry corruptions left its result byte-identical.
The pending final OpenADMET challenge result is excluded. The withdrawn projection report and closed-unmerged Lean Kernel Arena submissions are not accepted contributions. Tools without a separately evidenced research conclusion and documentation-only reports are not counted.