
Taming Performance Variability [pdf] - luu
https://www.usenix.org/system/files/osdi18-maricq.pdf
======
munin
This paper is cited by "Taming Performance Variability" but is probably worth
a read if you haven't read it before and find this paper interesting:
[https://www.seas.upenn.edu/~cis501/papers/producing-wrong-
da...](https://www.seas.upenn.edu/~cis501/papers/producing-wrong-data.pdf)

The authors of "Producing Wrong Data Without Doing Anything Obviously Wrong"
find a bunch of "optimizations" that don't do anything but change performance
by some percentage points. I think about this every time someone says big
companies care about "sub percentage points" optimizations - how much of that
is in the noise? Work like these papers ask these questions, which I think is
good. Computer scientists seem to have a worse time with empirical evaluation,
statistics, and experimental design. Not that any field of science does super
good at this all the time, but CS seems to have this as an "unknown unknown"
deficiency.

