About
Omega Protocol
Omega Protocol is Warren Smith’s independent research lab in Bristol, UK.
Omega checks what software, AI and scientific models actually do by building a separate test, then looking for ways that test could fail.
The work spans AI and its tools, mathematics, scientific models, software, and ways to measure performance.
Want the details?
How Omega checks its workExplore ↓
A check has to catch a relevant failure before a pass can mean something.
Research tools and computersExplore ↓
Scale, when a question needs it
Compute access is relevant only when a question needs it. Commissioning a resource is separate from producing a scientific result on it.
Omega holds an approved EuroHPC project on JUPITER, Europe's exascale machine, and has run verified jobs on its GH200 hardware.
Project EHPC-AIF-2026PG01-1357, Scaling and Reproducibility of Verifiable Multi-Agent Evaluation on JUPITER. Two commissioning jobs completed on the booster partition under that account on 22 September 2026, one on CPU and one on a GH200, both exiting 0. Commissioning runs only. No scientific result has been produced on JUPITER yet, and no allocation size is claimed here.
How results are describedExplore ↓
How claims are graded
Every result carries a grade recording how far the evidence reaches, and a statement of where it stops. The grade constrains the verb: a result graded MEASURED may not say “proves”.
- PROVEN
- Follows from stated premises inside a named frame. The frame and what has to be trusted are shown alongside.
- MEASURED
- Empirical, with a stated sample and method. The denominator is always shown.
- OBSERVED
- Witnessed in one instance, with no claim about how often it happens.
- OPEN
- A question posed and not answered.
A grade is a statement about the evidence, not about importance.
Current AI evaluation boundary. Omega has published bounded AI evaluation results, including checks on judges and agent controls. The broader AI evaluation layer remains partial: a corrected reference-defect replay and a multi-judge replay still lack evidence. Neither published cases nor this status imply general evaluator qualification.
Requirements, coverage and negative controls
A PROVEN result inside a small frame can say less about the world than an OBSERVED one, which is why the frame is shown next to it.
A coverage map tracks requirements, not evidence, and uses a separate set of words: holds, refuted (a counterexample exists), uncovered (nothing is checking it) and contested (deliberately never claimed). Contested is not a kind of uncovered, and the two are drawn differently everywhere on this site.
A pass is allowed to mean something only after the same check has been seen to fail on a case known to be bad. Each finding records whether such a control is present, none, not applicable or not established. No control is invented to fill the field.
CorrectionsExplore ↓
Corrections
A corrected value does not erase what it replaced. 3 corrections are on record.
The corrections
- Field
- open_unknowns
- From
- 7
- To
- 8
- Reason
- The figure was written from recollection instead of counted. The count is now derived from the records at build time, so authoring it is no longer possible.
- Was it published
- no; committed locally, never served
- Field
- capctl-iris.theoremCount
- From
- 29 (unqualified)
- To
- 29 at v0.1.3 (DOI 10.5281/zenodo.21505166); 40 at head dcc8ecb
- Reason
- The inventory was taken in July and not re-derived when the development grew. The count now carries its source commit, and the build refuses a theorem count without one.
- Was it published
- yes; both values are shown
- Field
- evidence.n-1.reproducibility
- From
- 13 of 13 re-derive from a public clone; this one does not
- To
- the minimal reproduction (omega-n1-permit-membership) re-derives from a public clone; the flagship bundles do not
- Reason
- A careful reader could read the page as saying the same artifact is both public and private.
- Was it published
- yes; both values are shown
External corrections that affect a cited result are shown on the result itself: the evaltrust repository was archived on 2 September 2026 after a dated correction, and every row citing it says so.
Who Omega is and what its status meansExplore ↓
About Omega Protocol
Omega Protocol is the independent lab of Warren Smith, working on his own in the UK. The findings on this site are his; the artifacts are public where the code permits it; and where a result has not been reproduced by anyone else, the site says so on the finding itself.
- What it is
- An independent research lab at an early stage, led by its founder. Omega investigates important technical questions about systems, models, experiments and claims, and checks the answers independently. The research programme is part of the work, and collaborations with domain experts are welcome.
- Institutional status
- One engineer, working independently in Bristol, UK, under the name Omega Protocol. There is no team, no funding round and no customer list, and nothing described here runs in anyone else's production system. It holds no accreditation and certifies nothing. 12 of its results have been accepted upstream by the maintainers of systems it did not build; that is review of a change, not endorsement.
- How it works
- Claim, subject, instrument, negative control, intervention, observation, verdict, limitation, reproduction. The loop is on the method page; every finding on the ledger carries those fields.
- How to inspect the work
- Each finding names its repository, its commit and the one command that produces the same numbers again. 20 of the 25 rows in this site ledger name a public repository and command. The current public audit separately verifies 46 public results and 40 checking-instruction sources. Where an artifact behind a case is not public, as with the flagship N-1 bundles, the page says so and why. Independent reproduction by others is recorded on every row, and is currently none.
- On the name
- The name is a name. Omega Protocol is not a published standard and has no adopters. The record format it began with is one artifact among those on the ledger.
Who is responsible
Founder & Principal Engineer
I am Warren Smith. I build the checks that say a system is behaving, then try to break them, including my own. The findings on this site are the result of that, and the ones where my own work failed are published in the same place as the ones where it held.
- Mathematical proofDevelopments in Lean 4, Z3 and TLA+, audited for hidden assumptions. A hardware proof merged upstream.
- AI and agent behaviourPermission boundaries, tool limits and multi-agent experiments, with live models and machine-checked cores underneath.
- Scientific computingIndependent recomputation, finite-difference validation and solver boundary testing on PyBaMM, PySCF and pandapower.
- Software investigationRunning two versions against the same inputs, planting defects to see whether a check notices, replay verifiers, and proofs that are then attacked.
- Measurement integrityWhether a judge or a score measures what it is being used to mean, and harness defects reported and fixed upstream.
Public repositories are at github.com/repowazdogz-droid. Contact is on the bring us a problem page.
TermsExplore ↓
Terms, in ordinary language
8 terms
- permit
- A small record saying which action was approved, on what, and when. Think of it as a signed note rather than a password.
- committed
- Written into an append-only record that cannot be quietly edited afterwards. A permit that was issued is committed; one somebody typed out is not.
- property
- One thing the system is supposed to guarantee, written precisely enough that you could tell whether it had been broken.
- coverage
- How many of those guarantees actually have something checking them. Low coverage does not mean the system is broken. It means nobody would find out if it were.
- counterexample
- A specific case that shows a guarantee does not hold. Not an opinion about the design, but a thing that happened.
- preregistered
- Written down and hashed before the work started, so nobody can quietly adjust what counted as success once the results are in.
- witness
- Something that measures what actually changed in the world, instead of believing the system's report that it succeeded.
- contested
- A requirement deliberately never claimed. Different from one that simply went unchecked, and drawn differently everywhere on this site.