Research / PyTorch, sparse CUDA matrix-vector product
Told to use deterministic algorithms, PyTorch produced 7 different outputs in 50 runs of the same sparse CUDA calculation.
ReproducedReported, no fix claimed- Result
- PyTorch was told to use deterministic algorithms, but the same sparse CUDA calculation produced 7 different outputs in 50 runs without raising or warning. The behaviour was reproduced across three PyTorch versions and traced to the cuSPARSE algorithm used by the sparse matrix-vector path.
- Boundary
- One GPU model (RTX 5090), the CUDA CSR matrix-vector path and one input. Reported upstream and not fixed; nothing is claimed about PyTorch’s deterministic mode in general.
- Who did what
- Investigated by OMEGA and reported by Warren in a comment on issue #202030. AI agents in OMEGA ran the experiments and read the source; Warren reviewed the evidence, reproduced the result and posted the report. The issue, and the matrix-matrix case it describes, were reported by another user.
- Public evidence
- Report: comment on issue #202030Issue #202030
The question
When PyTorch is told to use deterministic algorithms, does the same sparse matrix-vector product on a GPU return the same result every time?
PyTorch is a widely used framework for machine learning and scientific computing. Its deterministic mode exists so that a run can be repeated exactly, for debugging, testing and reproducible results, and where no deterministic algorithm is available it promises an error rather than a silent difference. A calculation that varies without that error looks reproducible and is not.
Reproduced: A reported behaviour was reproduced from a minimal, public input. Reported, no fix claimed: Reported to the maintainers with evidence; no fix is claimed.
What was tested
- Ran the same CUDA CSR torch.mv call 50 times in deterministic mode, with warn_only set so that a determinism warning would be recorded rather than raised, and counted distinct outputs byte for byte. The same calculation on the CPU served as a control.
- Repeated the count in strict deterministic mode on PyTorch 2.8.0+cu128, 2.11.0+cu128 and 2.14.1+cu130, all on one RTX 5090 (sm_120).
- Checked that the count can see variation and that deterministic mode can remove it: a planted source of noise, and index_add_, an operation deterministic mode does cover.
- Followed torch.mv into PyTorch’s source to the cuSPARSE call it makes, and checked whether that call site consults the deterministic flag.
- Called cuSPARSE directly on the same input, outside PyTorch, with the default algorithm and with CUSPARSE_SPMV_CSR_ALG2.
What happened
- On 2.14.1+cu130, 50 calls gave 7 distinct outputs and no determinism warning. The CPU gave 1.
- In strict deterministic mode, which should raise an error when no deterministic algorithm is available, no error was raised and the result varied on all three versions: 8, 8 and 9 distinct outputs in 50.
- The path appears to be addmv_out_sparse_csr, which calls cusparseSpMV with CUSPARSE_SPMV_ALG_DEFAULT without checking PyTorch’s deterministic flag. The fix already proposed on the issue does not appear to cover this path.
- Called directly on the same input, cuSPARSE varied with the default algorithm (6 distinct outputs in 50) and gave 1 distinct output in 50 with CUSPARSE_SPMV_CSR_ALG2 (cuSPARSE 12.6). NVIDIA documents ALG2 as deterministic for opA == NON_TRANSPOSE.
Challenge
The objections that would most easily undo the conclusion, and what answered each.
- Could the counting itself produce the variation?
- No. The CPU gave 1 distinct output in 50 on the same input, and deterministic mode removed the variation from index_add_ (20 distinct in 20 with it off, 1 with it on).
- Is it one PyTorch release?
- No. The result varied on 2.8.0, 2.11.0 and 2.14.1, in strict deterministic mode and without an error.
- Is the cause in PyTorch or in cuSPARSE?
- Called directly, cuSPARSE varied with the default algorithm and did not with ALG2, which NVIDIA documents as deterministic for non-transposed matrices. That points to the algorithm chosen at PyTorch’s call site without consulting the flag. No change to PyTorch itself was tested.
What survived
Did not survive
- With deterministic algorithms enabled, CUDA CSR torch.mv returns the same result every time or raises an error.7 distinct outputs in 50 on 2.14.1 with no warning; variation without an error on 2.8.0 and 2.11.0 as well.
Survived
- The CPU path gives a repeatable result on this input.1 distinct output in 50.
- CUSPARSE_SPMV_CSR_ALG2 gives a repeatable result on this input.1 distinct output in 50 (cuSPARSE 12.6), consistent with NVIDIA’s documentation for non-transposed matrices.
Not established, or not tested
- Whether ALG2 is suitable for the CSC or transposed path has not been established.
- One GPU model (RTX 5090, sm_120) and one input pattern were tested. Other GPUs, CUDA versions and ROCm were not.
- No change to PyTorch was built or tested. Whether ALG2 is the right fix, and what it costs in speed or accuracy, is for the maintainers.
- Half-precision CSR matrix-vector products gave 1 distinct output in 20 with deterministic mode both off and on, so the test could not tell either way and no conclusion is drawn about them.
Limits
One GPU model (RTX 5090, sm_120), PyTorch 2.8.0, 2.11.0 and 2.14.1, the CUDA CSR matrix-vector path and one input. The finding is about that path only: it is not a claim that PyTorch’s deterministic mode fails in general, and nothing was measured about any trained model.
Reproduce or inspect
- Source
- pytorch/pytorch/issues/202030#issuecomment-6444880180
- Revision
PyTorch 2.14.1+cu130 (release tested)- Command
python repro.py- Expect
More than 1 distinct output in 50 on an affected GPU, and no determinism warning (7 in the reported run on an RTX 5090); 1 on the CPU
The reproducer is given in full in the comment. The number of distinct outputs varies from run to run and between GPUs. The command is given as the repository documents it; it was not re-run for this page.