Case · Compilers and machine-learning infrastructure · Internal record

A compiled function returned a stale result, and the investigation never left the machine

PyTorch's TorchDynamo compiler, on a frozen reproduction of open issue #196127: an isinstance check against a runtime-checkable data-member Protocol is constant-folded without guarding the attribute it reads.

The system said: Compiled. No recompilation. The run completed.
The evidence showed: The compiled result belonged to an object state that no longer held; eager was right. Confirmed by an evaluator that recomputed both, on a machine nothing left.

§ 1

The claim tested

A function compiled with torch.compile returns what the uncompiled function returns, including after an object gains or loses the attribute the check depends on.

§ 2

What Omega did

Froze the task, the pinned PyTorch build, the evaluator and a hidden seven-cell matrix by hash before any inference.
Ran a locally hosted 27-billion-parameter open-weight model as the investigator, at temperature 0, with no network access and no message from a human or a frontier model during the run.
Had the evaluator recompute eager and compiled results and the recompilation counter for every case the worker submitted, and classify what drove any divergence, so the worker's own labels never decided the verdict.
Re-ran the hidden matrix after the run to confirm the phenomenon and to confirm that the evaluator rejects the near-miss variant where only the class changes.

§ 3

What was found

Two stale cases, both in the per-instance form the issue describes: one object gaining an instance attribute between calls, and two instances of the same class differing in that attribute. In each, the compiled function reused its cached result with no recompilation while eager gave the right answer.
Two controls that must not diverge did not.
The worker's hypothesis, that guards do not track per-instance attribute state for these checks, was marked as a hypothesis and is not credited as a root cause.
The worker also asserted, unscored, that the reverse call order recompiles correctly. The hidden matrix finds that order stale too. That claim is flagged and is not evidence.

§ 4

Verdict

CONTRADICTED: Compiled output matches eager output when a per-instance attribute changes. Two evaluator-confirmed cases on one pinned build; the phenomenon the open issue describes.
SUPPORTED: The investigation can be run on-premises by a local model against an independent oracle. On this one frozen task. Not a general statement about local models.

§ 5

What this does not establish

One issue, one pinned build, one run. Nothing about the inductor backend, unions, method-bearing protocols, larger programs or the second scored call order. The worker's own list of what it did not cover was empty, which overstates completeness; the untested dimensions are listed here instead. No root cause was verified in the compiler's source.

§ 6

The records

Every field of each finding this case rests on, as it appears on the ledger.

OBSERVED torchdynamo-compiled-result-stale-under-instance-changeverify · hidden trusted base A compiled PyTorch function returned a result computed for a different object state, with no recompilation, where the uncompiled function returned the right one. The investigation was carried out by a locally hosted model with nothing leaving the machine.

Claim: On a frozen reproduction of PyTorch issue #196127, torch.compile constant-folded an isinstance check against a runtime-checkable data-member Protocol without guarding the attribute it read, so a compiled function reused its cached result after an object gained an instance attribute, and gave the same stale answer for two instances of one class that differed in that attribute, while eager execution was correct in both. A frozen evaluator that recomputed eager and compiled results itself confirmed both per-instance cases and two controls, and a hidden seven-cell matrix re-run after the run confirmed the phenomenon. The investigating worker was a locally hosted 27-billion-parameter open-weight model at temperature 0 with no network access and no human or frontier-model message during the run.

Status: OBSERVED Witnessed in one instance. No claim about how often.

Question: Passed is not correct

Subject: PyTorch's TorchDynamo compiler on a frozen reproduction of open issue #196127 (third-party system); the harness, evaluator and controls are Omega's. third-party subject

Frame: One pinned PyTorch build; one frozen task; one local worker run at temperature 0, 48 of 100 turns used; a hidden seven-cell matrix withheld from the worker.

Method: The worker constructs cases and an eager-versus-compiled oracle; a frozen evaluator recomputes eager, compiled and the recompilation counter for every submitted case and classifies the driver of any divergence; design and evidence sealed by hash before inference and re-verified after.

Oracle: Eager execution of the same function on the same inputs, recomputed by the evaluator, never taken from the worker's report.

Negative control: Present Two control cases that must not diverge did not; the hidden matrix's class-mutation cell had to be rejected by the evaluator and was; 0 hidden-matrix tokens appeared in the worker's trajectory; both seals re-verified after the run.

Denominator: 2 evaluator-confirmed per-instance stale cases and 2 correct controls in the scored run; 7 of 7 hidden-matrix cells behaved as frozen; containment 47 of 47 checks.

Preregistration: Task, evaluator, hidden matrix and success bar frozen and hashed before inference; the bar was tightened from an earlier run before, not after, this one.

Limitation: One issue, one pinned build, one run. Nothing about the inductor backend, unions, method-bearing protocols, larger programs or the second scored call order. The upstream issue was already open and is not Omega's discovery; nothing was posted to it and no fix was proposed. The worker's own directional claim about which call order recompiles is unscored, conflicts with the hidden matrix, and is not evidence; its empty list of untested dimensions overstates completeness. No root cause was verified in the compiler's source. The result is a fact about this task, not a general capability of the local model.

Source: artifact not public

Reproduce: not reproducible from a public clone

Independent reproduction: None known.

This finding on its own page

All cases Bring us a problem