
State of the Art: Reproducibility in Artificial Intelligence [pdf] - capablemonkey
https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/17248/15864
======
sgt101
I think that the result is overcooked. Their hypothesis 1 is somewhat
falsifiable in that I don't think that there is a widespread reproducibility
crisis. I have been unable to reproduce results a couple of times in my
career, but I think that each time that was due to naughtiness (deliberate) on
the authors part or incompetence by me. Almost always you can reproduce and
when I have run into trouble I've found that the authors almost always help
out (most people are just delighted that you are interested!) On the other
hand this paper is very useful in that I think it will be used to establish
better criteria for papers in the future. I often reject papers because they
make no claim and have no results, contribution or conclusions (this makes
reviewing them quick so I really like papers like this !) I think that it
would be harsh to outright reject a paper because the hardware set up is
poorly documented, but it would be reasonable to ask for that change before
publication (for example). I agree with the authors that their criteria are
useful.

One issue though, open sourcing software is a good aspiration, but it's not
always possible due to IP and licensing - also export controls in some cases
(not always US -> other places too). If the community insists on opensource
pre-publication some important stuff is not going to get published.

------
sytelus
The paper claims that only few variables are documented so reproducibility is
low. I think that's not a good measure because this complexity has kept
increasing and it would require significant space in pdf to document every
little detail. The good measure is if paper comes with code and it reproduces
the claims. If I'm reading it right, this paper says AAAI and IJCAI had only
25% of papers with code. We really shouldn't be accepting AI papers without
code.

------
godzillabrennus
Reproducibility being difficult to impossible is not limited to A.I. research.

There is a reproducibility crisis in general in research of all fields.

[https://www.forbes.com/sites/quora/2017/02/09/how-the-
reprod...](https://www.forbes.com/sites/quora/2017/02/09/how-the-
reproducibility-crisis-in-academia-is-affecting-scientific-research/)

------
ohazi
That is one of the clearest abstracts I've ever seen in an academic paper.

~~~
ISL
[https://arxiv.org/abs/1110.2832](https://arxiv.org/abs/1110.2832)

I've seen some with half as many words, but can't find them in the time
alotted.

~~~
gm6001
[https://arxiv.org/abs/1807.11336](https://arxiv.org/abs/1807.11336)

