Jürgen Schmidhuber has become notorious in the deep learning community for his claims that new advances from recent years, such as Generative Adversarial Networks, should be credited to him because of obscure work he published in the 1990s.
At NIPS 2016, Schmidhuber interrupted a widely-attended workshop on GANs given by Ian Goodfellow, to make his argument (see [1] at 1 hour, 2 minutes). The community has generally taken Goodfellow's side. Just because Schmidhuber "had the idea" doesn't mean he should get the credit for other peoples' work decades later.
As any entrepreneur knows, just having an idea isn't enough. You have to put the work in, do the experiments and get results.
It has become a rite of passage. You have effectively made it as an AI researcher if Schmidhuber decides that your breakthrough is actually derivative work of an idea he published 30 years ago.
The crazy thing is ... he is not entirely wrong. Only mostly.
Yes, Schmidhuber could present things in a better way and be more charitable. But do not overlook his work -- he has genuinely groundbreaking contributions to early theory of intelligence.
I recommend having a stroll through his website, for anyone interested in AI (or even art, creativity, etc.):
Godel Machine : While clearly absolutely impractical, this is a nice theoretical construction on what we could mean by "self-improvement" in a real, definite way.
Speed Priors
Super Omegas (and various thoughts on algorithmic information)
Always keep in mind a grain of salt when reading his results, but they are genuinely great.
---
I think establishing causation and giving credit does have a role in research, namely giving additional opportunity to those researchers to do even better research, elucidating the history of science, and to an extent giving a monetary incentive and morale recognition for researchers w.r.t. prizes, grants, etc.
For example isn't the story of Einstein, or von Neuman, or Feynman, etc. part of a narrative that makes science compelling? They had very interesting and lively lives and opinions that sometimes drove their research tastes. There is value in preserving individual history (as well as disregarding it when not particularly relevant).
That said, a scientists (or a good citizen in general) should not be driven primarily by ego, and in science this becomes clearly bad, impeding open collaboration, exchange of ideas that are essential for good functioning and good progress of science; I'd say Schmidhuber does let that through sometimes, to the detriment of his work.
> because of obscure work he published in the 1990s
Well, did you really read it? In what respect do you consider it to be "obscure"? Because they used other terms to name the concepts?
> As any entrepreneur knows, just having an idea isn't enough.
You mean he should have patented it? Is this really the way to go: scientists in machine learning have to patent their work to make sure it's properly attributed in future?
I’m not familiar with the work in question, but this is a common theme in academic research. Sometimes there’s a kernel of an idea published by one individual which is later rediscovered independently (or not) and developed into a much more comprehensive theory. The latter is what the OP means by “just having an idea is not enough.”
> The latter is what the OP means by “just having an idea is not enough.”
That's quite far fetched given the sentence starts with "As any entrepreneur knows". Society can consider itself fortunate that every valuable scientific finding eventually prevails; but unfortunately - as many cases show - it often takes several publications of the same idea, and the respective earlier authors often fall into oblivion (if known at all).
With regards to his claim that he described a GAN before Ian Goodfellow did, it certainly seems like that is the case.
The problem is that, in his original Artificial Curiosity paper he talks so specifically about world-simulating reinforcement learning. He even uses the terms "World Model" and "Controller" instead of the more general "Discriminator" and "Generator." In this paper, he says the GAN is a more specific application of his Artificial Curiosity paper, but I see it the other way around.
Goodfellow should have cited the Artificial Curiosity paper though, as training two networks at odds with each other in parallel is explicitly mentioned by Schmindhuber in his original paper.
It also doesn't hurt that Bengio and others came out with research when you could actually run the damn thing. Schmindhuber's Artificial Curiosity paper doesn't list a single experimental result.
Schmindhuber perhaps should have shared in the Turing award, but at some point you have to cut it off. We stand on the shoulders of giants, and so do those giants. At some point, a giant is getting left out of the award.
I don't know enough about the history of this research to comment, but I want to comment on the more general theme of giving credit to those who deserve it in the computer science community. Specifically, Vojtěch Jarník deserves credit for what is commonly known as Prim's Algorithm.[0] In my opinion it should be called Jarník's Algorithm (I didn't come up with that, a lot of other people feel that way too).
He discovered it in 1930, decades before Prim (and also before computers). If he were British or American, and not Czech, my guess is his discovery would never have to have been "rediscovered." This is not to put down Prim, it's just let's give credit where credit is due.
I had the distinct pleasure to make a trip with late Nobel laureate Jack Steinberger to CERN. And while he was certainly deserving all the recognition he got, he himself mentioned that giving prizes like the Nobel to individuals (and I would presume Turing is similar) becomes increasingly outdated. When Nobel conceived the prize, it was still a common thing for individual researchers to make big advances more or less in isolation. That is just not the case anymore in almost any scientific field (math might be an exception, idk). Prizes should go to research groups because most fields of science have become so complex that its very hard to get anything done as an individual researcher. The nobel peace price is regularly awarded to institutions, so it is not inconceivable.
I don't understand the point of this, who did he want the Turing award to go to?
Is it the +30 people that he quotes in the text? Himself? No one?
ACM just acknowledged that DL was being used more and more than was allowing computers to do tasks that were out of reach before. They picked Bengio, Hinton and LeCun because they were are the forefront of that new movement which to me sounds valid?
When a Turing award was given for the Goldwasser-Micali encryption scheme the same argument could've been made with different names, that it was not the "first" public key encryption scheme that was provably secure because it was just the result of Diffie-Hellman? Should Merkle be credited?
Schmidhuber is salty because he believes he invented the ideas behind deep learning and has never gotten the proper credit. So, yes, he wants the Turing award to go to him.
It's not an argument without merit. His work on LSTM and related ideas were really important to the field. And, as with all heroes in science, the deep learning triumvirate of Bengio, Hinton, and LeCunn get most of the credit but were building on lots of existing work from researchers who don't get their due. But, as with all awards, you need to draw the line somewhere.
I understand that, I know he's not a nobody, but why draw the line at him? His own rebuttal shows that he was himself building on previous work and while LSTM are very important, SOTA models are phasing them out in favour of attention based models. In computer vision CNN are still king and as far as I know he has no claim to those.
On top of that (although I understand why a researcher might not consider this to be important), Theano really helped democratize model development and deployment and it was developed at MILA under Yoshua Bengio's. You can't just hand wave the tools that actually sparked the new movement.
I'm not going to comment on the politics of who deserves to win research awards, but as an aspiring researcher (PhD in progress) I do respect Schmidhuber's attention to detail in tracing the intellectual lineage of novel ideas. In my current research area, I find that most of the interesting observations and questions were posed in the early days of the field, and I am reluctant to attempt to publish something that simply "rediscovers" those concepts with a fresh coat of paint (figuratively) but doesn't actually make a meaningfully novel contribution.
I don't know who is right, but Schmidhuber certainly comes across as someone who is interested in an accurate history.
For example the backpropagation algorithm is often misattributed. After the Perelman story my faith in proper attribution in science is fairly low in general.
In a similar case Cooley and Tukey should also never have been given the Turing Award prize for Fast Fourier Transform, because Gauss apparently invented it already in 1805: https://www.cis.rit.edu/class/simg716/Gauss_History_FFT.pdf
But Gauss wrote his papers in neo-Latin and a very difficult notation so his findings went largely unnoticed. Also computers weren't available in the 19th century.
At NIPS 2016, Schmidhuber interrupted a widely-attended workshop on GANs given by Ian Goodfellow, to make his argument (see [1] at 1 hour, 2 minutes). The community has generally taken Goodfellow's side. Just because Schmidhuber "had the idea" doesn't mean he should get the credit for other peoples' work decades later.
As any entrepreneur knows, just having an idea isn't enough. You have to put the work in, do the experiments and get results.
[1] https://channel9.msdn.com/Events/Neural-Information-Processi...