This is one of the reasons I went into math and software. In math there is only black and white, provably true and patently wrong. Still, part of the reason this is so is that to show any results everything must be laid on the table. CS and Math papers are therefore open by design and the world has benefited. I hope the rest of science will move in this direction.
I'm guessing you have not read Proofs and Refutations by Imre Lakatos. It is worthwhile.
Also if you read on the history of mathematics, the various back and forth foundational issues raised by infinitesmals, Fourier series, proof by contradiction and the like are very interesting. Sure, it is possible for a modern mathematician to look at all that and say, They just didn't know how to do it, HERE you go. But over decades and centuries people were somehow successfully continuing to do math, even though they knew that something serious was wrong in their understanding.
If you can find a copy, George Berkeley's The Analyst was a cogent criticism of the foundations of calculus in his day. (He didn't have answers, but he identified real problems.) A variety of books were written by mathematicians in response. Uniformly these were much lower quality, and consisted of defenses of the foundations of mathematics, rather than acknowledgements that there were real issues. His criticisms were not taken seriously until that mathematical framework completely fell apart when Fourier series constructed "obviously impossible" things.
Today we are used to a very general notion of a "function". But historically a square wave simply wasn't a function. (Because if it was, how did you know how it interacted with infinitesmals?) And getting one out of the sum of a bunch of well behaved sins, with some simple integration, was a huge shock.
There are 'proofs' that contain errors, just as there are science papers that reach incorrect conclusions.
Also, it's not necessarily easier to verify a proof than it is to repeat an experiment (or perform additional supporting experiments), especially for complex proofs which very few people know enough to completely understand (eg. of Fermat's last theorem, or the Poincare conjecture).
Yes, that's why I no longer consider pursuing PhD in CS. It's a waste of life to reproduce non-reproducible papers that is full of buzzword.
Hopefully, more journals and confs will follow the Open Access and Reproducible Research model like IPOL.
> Each article contains a text describing an algorithm andsource code, with an online demonstration facility and an archive of online experiments. The text and source code are peer-reviewed and the demonstration is controlled.
Yes. Of the fields I've implemented algorithms from papers in, computer vision seems to be particularly adverse to discussing the (often critical) downsides of their algorithms. Often its something like "Camera calibration isn't exactly perfect? Well this scene reconstruction technique simply wont work."
> In math there is only black and white, provably true and patently wrong.
You'd probably be interested in Godel's incompleteness theorem.
Also, exact reproduction of other people's experimental work is highly unusual in CS, despite the theoretical possibility. Usually the materials and methods sections of papers in hard sciences like physics, chemistry, and biology are much more detailed, to the point of being recipes.
Finally, CS is not really a science. It lies somewhere between math and engineering, which are also not sciences.