the correct way to control randomness in scientific code is to have the RNG be seeded with a flag and have the result check out with a snapshot value. Almost no one does this, but that doesn't mean it shouldn't be done.
This is not correct on several levels. Reproducibility is not achievable in many real world scenarios, but worse it's not even very informative.
Contra your assertion, many people do some sort of regression testing like this but it's isn't terribly useful for verification or validation - but it is good at catching bad patches.
Did you read my post? I know what a seed is. Setting one is typically not enough to ensure bit-for-bit identical results in high-performance code. I gave two examples of this: CUDA GPUs (which do non-deterministic accumulation) and asynchronous threads (which won't always run operations in the same order).
Most scientific runs are scaled where you run multiple replicates. And not all scientific runs are high-performance in the HPC sense. Even if your code is HPC in the HPC sense, and requires CUDA, and 40,000 cores, you should consider creating a release flag where an end user can do at least single "slow" run on a CPU on a reduced dataset, in single threaded mode, to sanity check the results and at least verify that the computational and algorithmic pipeline is sound at the most basic level.
I used to be a scientist. I get it, getting scientists to do this is like pulling teeth, but it's the least you could do to give other people confidence in your results.
> consider creating a release flag where an end user can do at least single "slow" run on a CPU on a reduced dataset, in single threaded mode, to sanity check the results and at least verify that the computational and algorithmic pipeline is sound at the most basic level.
Ok, that's a reasonable ask :) But yeah as you implied, good luck getting the average scientist, who in the best case begrudgingly uses version control, to care enough to do this.
