

Evidence-based Software Engineering [pdf] - mblakele
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.113.571&rep=rep1&type=pdf

======
andrewcooke
the cornerstone of modern evidence-based medicine is the randomised trial.
trials can produce statistically significant results because there are many
humans and they are all quite similar. unfortunately, software projects are
both more diverse and less common. this is a critical difference: you simply
cannot replicate evidence-based medicine in the field of software development.

section 5.2.2 addresses some of this, but is then pretty much ignored in the
conclusion.

~~~
mblakele
Yes, that struck me too. Some developers do randomized trials of user
experience, but such trials aren't tests of development techniques - unless
perhaps you are working on an IDE.

And yet I think the points about negative evidence are relevant to software.
Whenever a hot new trend surfaces, it seems like negative early experiences
aren't reported. Later on there is usually a backlash, but it would be in our
interest to shorten that cycle.

------
vanni
According to HN Guidelines (<http://ycombinator.com/newsguidelines.html>),
next time please append _[pdf]_ to the title. Thank you.

~~~
mblakele
Done, thanks.

~~~
vanni
Well done. :)

------
mblakele
This paper is from 2004, but I don't think I've seen it discussed here before.
Any experiences or insights to share on this topic?

~~~
spenrose
Greg Wilson did a book, which I did not enjoy, on this topic:
<http://shop.oreilly.com/product/9780596808303.do> and a slideshow, which I
recommend: <http://www.slideshare.net/gvwilson/bits-of-evidence-2338367> .

My take is that there is much to learn from science about _how to evaluate
propositions regarding software engineering_ (most, but not all, of them are
unsupported) but few new useful ideas.

Another reference along these lines: [http://www.amazon.com/Facts-Fallacies-
Software-Engineering-R...](http://www.amazon.com/Facts-Fallacies-Software-
Engineering-Robert/dp/0321117425)

------
akkartik
Is table 1 really just doing a search-and-replace of 'clinical' for 'software
engineering'??

