Research / Chemprop 2, command-line training
Three training settings were silently ignored unless the loss function was also named.
ReproducedFix submitted, not yet merged- Result
- Three training settings were silently ignored unless the loss function was also named. The fix also covers saved configuration files and the tested hyperparameter-search path, and is submitted for review, not yet merged.
- Boundary
- Command-line training and one hyperparameter-search trial on macOS (CPU); not the search algorithms themselves, and no model accuracy was measured.
- Who did what
- Investigated and fix submitted by OMEGA (pull request #1417). AI agents in OMEGA wrote the patch and tests and ran the checks; Warren chose the scope and approved the submission. Issue #1416 was reported by another user.
- Public evidence
- Pull request #1417 (under review)Issue #1416
The question
Do Chemprop’s loss settings take effect when the loss function is left at its default?
Chemprop predicts molecular properties from chemical structure and is used in chemistry and drug-discovery research. Its uncertainty-aware training modes take settings such as the width of a prediction interval. If a setting is dropped, the run still completes and the model looks normal, but it was trained for a different objective from the one requested.
Reproduced: A reported behaviour was reproduced from a minimal, public input. Fix submitted, not yet merged: A fix was submitted to the maintainers with its evidence. It is not merged or released.
What was tested
- Trained each affected task type with each setting, on the main branch (db0bb86) and with the fix, and read the loss function back from the saved model.
- Re-trained from each saved configuration file, and ran one hyperparameter-search trial, to check that a setting survives both paths.
- Added 14 test cases, including runs that give none of the settings as controls, then broke the fix deliberately in three ways to check that the tests notice.
What happened
- On main, --alpha 0.05 trained with 0.1 and --v-kl 0.5 trained with 0.2, across three task types. The settings took effect only when the loss function was named explicitly.
- The hyperparameter-search path had the same defect: a trial given --v-kl 0.5 trained with 0.2 before the fix and with 0.5 after it.
- 6 of the 14 new test cases fail on main and all 14 pass with the fix. Runs that give none of the settings behave exactly as before.
Challenge
The objections that would most easily undo the conclusion, and what answered each.
- Does the fix change runs that never used these settings?
- No. Control runs with no settings produce the same loss before and after, and they are part of the tests.
- Could the new tests pass without the fix?
- No. Each of the three deliberate breakages makes at least one test fail.
What survived
Did not survive
- Loss settings given on the command line reach the default loss function.Ignored on main for three task types, read back from the saved model.
Survived
- Runs that give none of the settings are unaffected.Same loss before and after in every control case.
Not established, or not tested
- Two questions about intended behaviour are left to the maintainers: which value --v-kl should default to, and how configuration files saved by older versions should be treated.
- The full test suite, GPU training and atom- and bond-level hyperparameter search were not run.
Limits
Command-line training in Chemprop 2 on macOS, CPU only. 183 neighbouring tests pass with the fix. No prediction accuracy, research result or drug-discovery outcome was measured.
Reproduce or inspect
- Source
- repowazdogz-droid/chemprop/tree/fix-loss-hparams-default-criterion
- Revision
09a2219d- Command
python -m pytest --no-cov -q tests/cli/test_cli_regression_mol.py::test_train_loss_hyperparameters tests/cli/test_cli_classification_mol.py::test_train_dirichlet_v_kl tests/cli/test_cli_MAB.py::test_train_evidential_v_kl tests/cli/test_cli_regression_mol.py::test_train_config_replays_loss_hyperparameters tests/cli/test_cli_regression_mol.py::test_train_legacy_config_warns- Expect
14 passed
Run from a Chemprop checkout of the pull request branch with its test dependencies installed. Re-run by OMEGA at this revision on 18 October 2026 (macOS): 14 passed.