I think it should be on a desktop class machine. I'm less concerned about absolute numbers and more with variance. Laptops generally cannot dissipate all the heat from full CPU usage for more than a minute or two. Benchmark runs, if they exercise the CPU hard for any length of time, get thermally limited and the results end up with substantially more variance.
Even things as simple as build times, I've seen vary by 20+%.
Benchmarking is always a hard problem - no such thing as spherical chickens in a vacuum. That said we'd love to at least standardize the setup for other people running the benchmarks; that is why we opened all the code and data, as a starting point.
For the study, measurements were run with one of our personal laptops (an X1 Carbon from 2016 with an i7 U6600 CPU and 16 GB of Ram, which is indeed a pretty powerful machine). We tried very hard to limit the impact of frequency throttling due to limited thermal dissipation of the device for the long-running benchmarks. In fact, for the measurements we put the laptop outside at 0 degrees Celsius, and we could observe that the CPU temperature did not go beyond 60 degrees (which is pretty low).
Do you have any suggestions on how we could improve this setup? We welcome all contributions.
Even things as simple as build times, I've seen vary by 20+%.