
Nontrivially fillable gaps in published proofs of major theorems - mathgenius
https://mathoverflow.net/questions/352249/nontrivially-fillable-gaps-in-published-proofs-of-major-theorems
======
btrettel
In my experience, Linus's law [1] is false in scientific research. There's a
popular view that peer review is a strong filter that allows other researchers
to have confidence that a claim is correct. My guess is that this might
contribute to Linus's law being false, because many researchers actually do
seem to think "it's been checked, so it must be okay, I don't need to check
anything myself". Or, perhaps, people don't do much checking in the first
place.

My view is that peer review is a fairly low (and random) bar to pass that
doesn't necessarily say much about the validity of a work. More people should
check things more carefully.

Recently I got reviews for a paper I wrote back, and a reviewer was skeptical
of one claim I made because (paraphrasing) "If this were true, that would mean
all previous researchers were wrong." They're exaggerating as it would only
mean many researchers were wrong; I think some were skeptical of this for a
long but kept quiet about it. But the idea that "everyone seemed to accept it,
therefore you are probably wrong" independent of any arguments made seems
popular, despite it being obviously fallacious. I have some arguments. The
reviewer should engage with those arguments rather than make some appeal to
popularity. Edit: To be clear, what I was claiming here would overturn
something dating back to the 1930s that last received a well-accepted revision
about 40 years ago.

[1]
[https://en.wikipedia.org/wiki/Linus%27s_law](https://en.wikipedia.org/wiki/Linus%27s_law)

~~~
ColanR
I've been seeing discussions about the need for more transparent peer review,
and more replication. It strikes me that if the peer reviews of an article
were published with the article, like the commentary next to court cases in a
law textbook, we would have a much more accurate idea of exactly what the peer
review validated about the published article.

~~~
skat20phys
So, I've edited a journal, been on journal editorial boards, and reviewed a
lot. I'm skeptical that more transparency in peer review will make a
difference at all. In fact, some of the proposed solutions might hurt a bit.
In the very least, I think there's a tidal wave of change in publishing that's
making traditional academic peer review moot.

The linked question is interesting for me to think about given one of my
relatively recent experiences (this is all in the area of statistics, so kind
of directly relevant). I was asked to review an article, and as a counterpoint
to one of my concerns, the author cited an in-press paper in a fairly well-
respected journal. So I go to look at it, and it makes little sense to me, in
that it contradicts a bunch of other things that are known. I look at it
closer, and it turns out there's a subtle but important notational error
carried through much of the proofs that basically invalidates the whole paper.

So I contact the editor of _that_ journal to feel out the response to writing
a commentary about the issue, with a fairly detailed explanation of the flaw
in the proofs. Instead of being receptive or at least neutral, the editor
throws up all sorts of obstacles — not exactly threats, but strong
discouragement in the form of a long list of criteria that had to be met,
several of which were completely unnecessary. My colleagues (who are also
editors of other journals) got even more upset than me, believing that the
editor was trying to bury the error and so forth. One even threatened to
expose the exchange on twitter or something.

The truth is, I don't know that I cared that much about this particular topic
to really put the effort into writing a commentary, fighting with the editor
to publish it, and skewering the author's ego in the process. My friends and I
discussed just putting the commentary on an online archive, but it wasn't
clear anyone would make the connection with the paper. Also, for unrelated
reasons, I shortly afterward wrote a different paper that was more
comprehensive in scope that sort of superceded that flawed proof anyway (that
is, if someone read this paper of mine, the results of the first erroneous
paper would probably seen as as irrelevant).

In this case, the flawed proof/paper _was_ peer reviewed. Making the reviews
transparent wouldn't have mattered because in the end the paper was published.
Maybe one of the reviewers raised the issues but it was published anyway, so
published reviews would tip a reader off? But at that point where are we? If I
_had_ done something (and maybe I still will?), I would have just published it
in a public academic archive anyway. What, then, is the point of peer reviewed
journals? Are we better off just posting papers publicly, and publicly
commenting on them? Is stripping anonymity from reviewers a good or bad thing?
Won't that discourage rigorous review, for fear of repercussions against
reviewers? Is review really all that rigorous anyway?

My personal impression is that the volume of academic publishing has increased
so much that it's impossible for readers to really keep up, and making it more
difficult for scientific consensus to form completely. Publicly available
papers in archives is a natural extension of this. What this means is that
readers increasingly pick and choose which literature they read and cite,
which truth they want to reinforce, and what truth they want to suppress. When
you open up peer review to be public, those reviews become just another part
of that literature. The peer reviews become blog and twitter posts, which is
_absolutely fine_ , but then a reader just selectively picks and chooses which
what reviews and blog posts they cite, and so forth and so on.

For the record, I'm very much for open publishing, and open discussion of
literature. I just think that academics hasn't wrestled with the implications
of that, in terms of what it will look like (e.g., amplifying fads, decreased
signal to noise ratio, increased feedback loops), and whether it's worthwhile
to vigorously maintain anonymous peer review to have that as another form of
literature evaluation. There's already a lot of public discussion of papers,
and this will only increase regardless of what happens to peer review. Maybe
the issue is who does the anonymous review? Maybe just opening up papers to
anonymous commentary is the right way to go?

~~~
btrettel
> My personal impression is that the volume of academic publishing has
> increased so much that it's impossible for readers to really keep up, and
> making it more difficult for scientific consensus to form completely.

I've noticed this as well, and it just makes me think that quality reviews
become more important over time.

Unfortunately, in my experience most reviews basically mirror what a couple
recent reviews said, adding a few new papers. This is assumed to be up-to-date
when in fact if the older reviews missed some important older papers, it's not
up-to-date. And that's what I see: important papers missed by reviews in the
past continue to be missed. I don't know if this experience is valid outside
of fields other than my own, however. In my PhD I've tried to comprehensively
review the literature and I've found quite a few important missed papers.

I think few people actively pick "which truth they want to reinforce, and what
truth they want to suppress." My impression is that literature reviews are
done more out of convenience than an intentional desire to distort the
literature.

> My friends and I discussed just putting the commentary on an online archive,
> but it wasn't clear anyone would make the connection with the paper.

I think PubPeer is designed for this situation:
[https://pubpeer.com/](https://pubpeer.com/)

~~~
skat20phys
I knew there were things like what I was mentioning, but couldn't remember the
names of them. Thanks.

I don't think any of these biases are necessarily consciously enacted, but I
think they exist in some of the ways you mention. There's a sort of echo
chamber effect or positive feedback loop with citations.

Somewhere I remember reading a bibliometric analysis of citation patterns in
the nutritional sciences regarding the effects of salt. There were two huge
clusters of papers, one basically "salt is basically fine" and the other is
"salt should basically be avoided". The papers in a cluster cited each other a
lot, and not so much papers in the other clusters.

I agree that quality reviews become more important over time, but there's the
issue of "according to who?" I suppose this becomes an expert judgment call
and is the nature of these things, as it always has been, but I feel like as
things become more parochial and balkanized the meaning of a "good review"
changes somewhat, or becomes harder to agree on.

------
enricozb
Hopefully as theorem provers improve and become more expressive, we can begin
to state these theorems and even formalize proofs of them, letting a computer
verify the proof for us. Check out Lean[1] if this sounds interesting to you.

[1]: [https://leanprover.github.io/](https://leanprover.github.io/)

~~~
gpm
I've been checking out lean in my spare time over the last week.

While it seems like a nice formal proof checker, it doesn't seem to have any
automated their theorem proving abilities yet?

I can't imagine trying to do math where I have to manually supply a proof of
every trivial statement... Something like Isabelle and it's sledgehammer
automated proof finder seems more reasonable to me (which I hope to find time
to dip my toes into this week).

Am I missing something?

~~~
bonoboTP
I assume it could be done by having a "library" of such trivial statements.
It's like in programming: we wouldn't want to deal with trivialities like how
to decode utf8 or deal with files in a high level app, so we use
libraries/APIs. You have to memorize them to some extent, but there can be
autocomplete, documentation etc, just like in programming. Surely it needs a
different mentality compared to old fashioned idealist type math people who
may dislike such mechanization as taking the "soul and art" out of pencil and
paper math, but I think the new generation of math people do have more overlap
with CS-like thinking and they would be onboard.

~~~
gpm
Trivial statements of the kind I'm talking about are often incredibly proof
specific, because they are often of the form "take the proof context and
rephrase it so a pre proven theorem applies". Not the type of thing that can
be enumerated in a library.

~~~
bonoboTP
Well, I'm not a mathematician, but it seems like if it's really trivial, it
shouldn't be hard to formalize on a high level why it's true, like almost in
natural language and then compile that high-level expression into a lower
level argument. And if it's hard to even do that, then is the statement really
trivial, or is it perhaps something that seems intuitive, but may not be sure
(perhaps some pathological edge case may apply)?

~~~
gpm
So here's a dumb example I just came up with.

Let's say we know that x = 3^a 5^b. It's a trivial fact at this point that x
isn't divisible by 2 (by the fact that prime factorization is unique).

Disclaimer: I'm not actually good enough with lean to prove this sort of
statement quickly... I could be under or even over selling lean here, hence
why I phrased my original comment as a question about whether I was missing
something.

If I was trying to prove this formally, I'd have to say something like (in
computer speak): Suppose x is divisible by 2, x = 2 * y, y has a prime
factorization p by <theorem in library>, and 2 * p is therefore a prime
factorization of x (ouch, already not sure how to specify that statement). 2 *
p = x = 3^a 5^b, so 2 * p = 3^a 5^b. Both are prime factorizations, prime
factorizations are unique by <theorem>, therefore both expressions should have
the same number of 2 terms, but the one on the left has at least one, and the
one on the right has 0, so they don't. This is a contradiction, so x is not
divisible by two.

You see why I'd like a computer to fill in the long formal proof instead of
doing it myself? It's not because the statement might be false, it's because
it's a pain to phrase everything in terms of the re-usable result about unique
prime factorization.

While I'm disclaiming things, disclaimer 2: I'm a hobbyist, not a real
mathematician.

~~~
bonoboTP
I didn't mean that the necessary high level tools exist today. It may very
well be that computerized formal proof systems are at the stage where
programming was when only assembly existed. It was very tedious to express
what you wanted. I think the analogy is quite close:in both cases you're
trying to express an idea very precisely, for an uncompromisingly literal-
minded computer.

I was formulating a vision with ergonomic tooling, helpful language constructs
that haven't been invented yet, etc.

~~~
gpm
But that big paragraph _is_ the natural language version of the formal proof.
It's less strict and has more helpful concepts than any programming language
in the world. It's that big because there actually are a lot of details needed
for the proof. For instance it's not necessarily true that x = y = z means x =
z, I need to provide a proof of that (transitivity of equality), it's not
necessarily true that just because x = 3^a 5^b doesn't mean x isn't also 2 *
<some other prime factorization>, I need a proof of that. Etc. Unless the
computer is smart enough to do all _that_ work for me. In which case the
language isn't just nicer, it's also got an automated theorem prover.

Something like lean isn't assembly, it's a full fledged language with
abstractions, better IDE support than I've seen for any "real" programming
language, the ability to program new "tactics" (methods of proof), and so on.
I'm not trying to prove things in raw logic in it. The thing is I don't want
to be trying to prove some of these things at all, I want the reader (the
compiler) to just go ahead and supply their own proof.

------
beleuze
related: Epistemic Injustice in Mathematics (2018)

Rittberg, Colin Jakob and Tanswell, Fenner Stanley and Van Bendegem, Jean
Paul.

[http://philsci-archive.pitt.edu/15133/](http://philsci-
archive.pitt.edu/15133/)

------
downerending
Better: Gaps not trivially fillable...

That is, the point is that the gaps cannot be trivially filled.

------
chisleu
In the south, I don't think we really did. It evolved, sure, but the southern
accent is based on southern english royalty.

~~~
seanhunter
I think the parent comment relates to a different subject.
[https://news.ycombinator.com/item?id=22288440](https://news.ycombinator.com/item?id=22288440)
specifically.

