
Deep Learning: Our Miraculous Year 1990-1991 - eugenhotaj
http://people.idsia.ch/~juergen/deep-learning-miraculous-year-1990-1991.html
======
tlb
I encourage reading this, not as self-promotion, but as a first-person history
of what it feels like to be too early with a technology.

Someone out there is probably experimenting with something world-changing, and
has all the ingredients except for a few more iterations of Moore's Law. It
would feel a lot like working on deep learning in 1990. If you think you might
be on this path, it's worth studying the history.

~~~
throwawayjava
Do we have a few more iterations of Moore's law?

~~~
Agebor
Even if we don't, the progress is not going to stop, for example on:

\- lowering the price of each chip - you can get that by more automation.

\- lowering the cost of energy used by a chip - you can have that by raise of
renewable energy generation and its decentralisation (and again, more
automation).

The point is that automation caused by AI will start a reinforcing feedback
loop where more and more work can be done more cheaply, speeding up automation
itself too.

~~~
Barrin92
>The point is that automation caused by AI will start a reinforcing feedback
loop where more and more work can be done more cheaply, speeding up automation
itself too.

there isn't much evidence that AI has accelerated the rate of automation, and
people have been saying this about information technology for the last 4
decades already. By any account, automation and growth contribution of the
technologies are low by historical standards.

The primary mechanism that has kept Moore's law alive up until now is
miniaturization of transistors and we're going to run into a wall on that
front pretty soon.

~~~
K0SM0S
I will not counterargue your main point because this is indeed a matter of
debate from a 'technical' standpoint.

However, in broader economic terms, I think the idea that AI may 'accelerate'
the world in general is largely indirect: for instance, by saving time and
money in other areas of life (because better tools, cheaper means, infra,
etc), people become more able to perform _their_ job. There are obviously
diminishing returns to such optimization, as to any natural/economic process.

~~~
legulere
Automation also often means that useful jobs get turned into bullshit jobs,
that stay there e.g. for political reasons, sometimes leading even to
decreased efficiency.

------
pjbk
I guess many institutions and research groups can write similar accounts. Even
the late 80s were somewhat productive concerning NNs and what today we call
ML, just by searching publications of that era.

We had also some relatively sophisticated tools, and looking back in time one
could say they were deep-learning-ish. In my personal case I did some research
for weather forecasting using BPN/TDNN, Kohonen and RNNs with the Stuttgart
Neural Network Simulator [0]. It allowed some flexibility creating and
stacking models.

[0] [http://www.ra.cs.uni-
tuebingen.de/SNNS/welcome.html](http://www.ra.cs.uni-
tuebingen.de/SNNS/welcome.html)

------
shmageggy
God, Schmidhuber is insufferable.

This whole account has virtually zero mention of how later techniques improved
upon or innovated on his, and very little account of how his contributions
were (like everyone else's) evolutions of existing methods. It reads almost
like Schmidhuber or his students invented and solved everything from scratch,
and nobody else has done shit since.

The guy clearly wants to be more included in the standard narrative, but being
so self aggrandizing is doing him zero favors. If were capable of writing an
honest, charitable account of how his work fits into a much larger field, it
would be much easier to take him more seriously.

~~~
mjn
I mean it's self-promotional yes, but I read this as more of a blog post about
advances specifically in his own group. For the Schmidhuberian take on the
broader history of deep learning, this other one's the go-to article (though
it's much longer):
[https://arxiv.org/abs/1404.7828](https://arxiv.org/abs/1404.7828)

Not everyone likes that article either, but it does at least extensively cite
prior work, i.e. accounts for "how his contributions were (like everyone
else's) evolutions of existing methods". In particular, sections 5.1–5.4
credit a large amount of work from the 1960s-80s that he considers
foundational.

------
plmu
One of the early applications was pattern matching for LHC. I was in one of
the groups in which some (not myself) worked on this and put the neural
networks, using the just developed theory, in hardware with FPGA's.

After a few years the three (post-docs) left and founded a startup. I lost
contact with them. I think they were too early for broader applications, and
had left the field completely in the early 2000's, when it really took of.

Here is a book that the author of the referenced article , and the people from
my group (Utrecht University), contributed to:
[https://link.springer.com/book/10.1007%2F978-1-4471-0877-1](https://link.springer.com/book/10.1007%2F978-1-4471-0877-1)

------
alexcnwy
1989/1990 was also when convolutional networks first started working with
LeCun’s breakthrough paper on digit recognition.

Incredible to think how much amazing research was happening back then and
wonder what research is being done now that will change our lives in the next
30 years.

------
bonoboTP
> In surveys from the Anglosphere it does not always become clear [DLC] that
> Deep Learning was invented where English is not an official language.

Even if you disagree with Schmidhuber's assessment of his own importance, I
think this is clearly true.

There is a certain arrogance (or not-invented-here syndrome) in the
Anglosphere (or North America) towards research done elsewhere.

------
nafizh
It was a travesty Schmidhuber didn't receive the Turing award along with
Hinton, Lecun, and Bengio last year.

~~~
bonoboTP
It does seem to me that there could be some bias in this award's history.

The Turing Award has been awarded every year (usually to multiple people)
since 1966.

Look it up on Wikipedia. How many laureates of the 70 can you find who
performed their research outside of the Angloshpere? I didn't look in detail,
but after a quick glance it seems about 5 out of the 70 (Daal, Nygaard,
Shamir, Naur, Sifakis)? (Or how many who grew up outside the Anglosphere?)

Maybe that reflects the true state of things and almost all of CS was
developed in the Angloshpere. Even if that's so historically, I think it may
induce some bias when evaluating people's contribution from outside the Anglo
community and network.

------
KKKKkkkk1
It seems that Schmidhuber is claiming credit for deep learning and is
implicitly comparing himself to Albert Einstein. How accurate is his
assessment?

~~~
goldemerald
My goal is to one day have Schmidhuber angrily claim that my research was done
by him in the 90s like what happened to Ian Goodfellow [0].

[0]
[https://www.reddit.com/r/MachineLearning/comments/5go4sa/n_w...](https://www.reddit.com/r/MachineLearning/comments/5go4sa/n_whats_happening_at_nips_2016_jurgen_schmidhuber/)

~~~
account73466
Schmidhuber was more right than wrong

~~~
dgacmu
Interesting that you're a new account that has only ever posted in this thread
in defense of Schmidhuber.

~~~
account73466
Must be Schmidhuber then. I am not but I have thousands of DL citations on my
name.

------
2sk21
As an old-timer in neural networks, this was interesting. However I should
note that we did not call it "deep learning" back then. It was simply "neural
networks".

As I write this, I am looking at the book "Parallel and Distributed
Processing", (with the blue cover) an edited compilation of papers on neural
networks published by the MIT Press in 1987. I myself spent the summer of 1990
implementing the back-propagation algorithm as described in chapter 8 of this
book which is entitled "learning Internal Representations" by Rumelhart,
Hinton and Williams.

I myself got my PhD in 1992 for coming up with an algorithm for speeding up
back-propagation when the training set is imbalanced.

An Improved Algorithm for Neural Network Classification of Imbalanced Training
Sets. November 1993IEEE Transactions on Neural Networks 4(6):962 - 969

------
jumpingmice
The prominent developers of deep learning techniques within google were quite
upfront that they were applying old techniques that had not been practical
until massive datacenters expanded the parameter space and training power.

~~~
nullc
Have they been equally upfront in their patent applications?

------
kerng
This is pretty cool. Always interesting to see how things eventually become
mainstream whereas origins go back decades, sometimes more.

