
Geoffrey Hinton has joined Google - roquin
https://plus.google.com/u/0/102889418997957626067/posts/GWe4AscQdS7
======
srean
HN'ers may have a good laugh at these taken from Yan LeCun's page. LeCun made
it a tradition to have Hinton jokes in the lines of Chuck Norris ones (or more
appropriately Doug McCllroy ones).

A few will recall that neural networks all but died from the US after Minsky's
damning book. Hinton gave backpropagation which is one of the foundational
pillars of feed-forward neural net algorithms. With his new thrust on whats
called "deep belief networks" he is challenging his own early seminal
contribution in the field. Not often do you see researchers throw away such
huge swathes of their own work and start again to solve the same problem.
Unless you are Niklaus Wirth of course.

Some background is necessary to get the inside jokes, but I have tried to
minimize the requirement.

    
    
        Geoff Hinton doesn't need to make hidden units. They hide
        by themselves when he approaches.
    
        Geoff Hinton discovered how the brain really works.
        Once a year for the last 25 years.
    
        Markov random fields think Geoff Hinton is intractable.
    
        Geoff Hinton can make you regret without bounds.
    
        Geoff Hinton doesn't need support vectors. He can
        support high-dimensional hyperplanes with his pinky.
    
        All kernels that ever dared approaching Geoff Hinton woke up convolved.
    
        The only kernel Geoff Hinton has ever used is a kernel of truth.
    
        After an encounter with Geoff Hinton, support vectors become unhinged
    
        Geoff Hinton's generalizations are boundless.
    
        Geoff Hinton goes directly to third Bayes.
    
    

Links: <http://yann.lecun.com/ex/fun/index.html>

<http://en.wikipedia.org/wiki/Backpropagation>

<http://en.wikipedia.org/wiki/Perceptrons_%28book%29>

<http://en.wikipedia.org/wiki/Niklaus_Wirth>

Re: Downvotes. I seem to have touched a nerve. Yes I agree humor is frowned
upon here, and I largely agree with that, but when its not humor for humors
own sake, I sometimes make an exception. Not everyone would know who Hinton
is, but may gauge that he is someone important from these fun anecdotes. But
of course you are free to like it or dislike it, no hard feelings either which
way.

~~~
IanCal
I like the quotes, however:

> With his new thrust on whats called "deep belief networks" he is challenging
> his own early seminal contribution in the field

I don't know if I agree with that, he still uses backprop. Backprop has
_always_ been known to have problems when you scale to millions of
connections, and his work on RBMs/DBNs is really quite old. What was novel
more recently was showing that the contrastive divergence step need only be
performed once, rather than 100 times, while the performance remained similar.
The networks are generally still 'fine tuned' with backprop.

Still, the focus on generative networks (not sure if that's the right term
still, been a while) and single layer training is fairly recent even if the
concepts are quite old.

~~~
srean
Mostly agreed, one difference that I would like to highlight is that errors
are not always backpropagated across all the layers. In addition to
contrastive divergence the breakthrough has been that you can get away with
unsupervised learning (like with autoencoders) in the layers.

On the comment that RBMs are new, now I have to come to accept that if one
looks hard enough almost all things are old, only the name changes !

------
kolektiv
Sigh. I don't know whether I'm alone in being saddened that many of the
excellent thinkers of our time, who once might have spent their lives probing
the the very limits of our comprehension in their field, now end up
comfortably turning their minds to the services of (in the long term) rather
short sighted commercial ends.

I can imagine that this sentiment is probably against the prevailing HN mood,
but I've been thinking a lot lately about how certain kinds of thought and
investigation are only enabled and supported by certain structures. The point
of a university used to be being the highest pinnacle of thought. A place
where people could have everything not related to intellectual pursuit taken
care of so they could devote themselves to the pursuit of greater knowledge.
Now that seems to be replaced by a corporate campus.

It's a cliche, of course, but how many breakthroughs of meaning are we missing
out on because the brightest and best of our generation are now no longer
seeking after truth, but instead seeking after a way to make more people click
on an ad? Ah well, it's late, and as ever I'm post-sober. Best of luck to him
and his team - I've enjoyed his lectures and papers. I hope they may continue,
but I sadly suspect not.

~~~
ok_craig
The common lament about Google being just a place that optimizes ad clicks is
overused, and lazy thinking.

I don't even think it's a conclusion that's the result of actual thought
anymore, just reverb.

Google is a big company. It would not even be worth it or effective to have
every single engineer dedicated to this ad click business. There's a team for
that, and there are lots of other teams, for lots of other interesting things.
Many of them do serious intellectual research.

My impression of Google's leadership, employees, and choice of projects often
leads me to believe that they sell ads so they can continue making really cool
stuff - not that they try to come up with cool stuff so that they can sell
ads.

I also disagree with the whole premise that goal-oriented or profit-driven
intellectual study is in some way less pure, less worthy, less boundary-
pushing, or more short-sighted than purely academic study. How does the
initial incentive for a thought affect whether or not the conclusions from
that thinking are new knowledge about our universe? It doesn't.

As long as he's not sacrificing what he does to become a paper pusher, which
he isn't doing, I don't see how it's different. And I think ultimately the
closer his study is to being manifested in real things that we use, the faster
his knowledge will be refined towards the path of truth, and the faster
humanity will actually learn and benefit.

~~~
wtallis
Google simply hasn't been around long enough to earn the kind of credibility
that Bell Labs and Xerox PARC had.

~~~
paulsutter
So when is the last time that PARC or Bell Labs produced anything of note?
Seriously. 15 or 20 years?

You can only rest on laurels for so long before they dry up and blow away.

Google meanwhile is an incredible company. Are you saying you know better than
Hinton where the real progress is being made?

~~~
Homunculiheaded
I think most people considered the spin off into Lucent to be the true end of
what Bell Labs was. There hasn't effectively been a Bell Labs since 1997.

~~~
Daniel_Newby
The true end of Bell Labs was when Lucent started financing customer
purchases. They were carrying dot com bubble debt on their books as if it were
a solid asset. When the music stopped they had spent their actual assets to
the brink of destruction.

------
cs702
Something really interesting must be happening with AI at Google, because in
the past few months both Ray Kurzweil (the best-known proponent of the
singularity) and Geoff Hinton (the crazy-talented individual who invented
deep-belief networks using interconnected "restricted Boltzman machines"[1])
have joined the company.

\--

[1] For an overview of deep belief networks, see these videos:
<http://www.youtube.com/watch?v=AyzOUbkUf3M> ,
<http://www.youtube.com/watch?v=VdIURAu1-aU> , and
<http://www.youtube.com/watch?v=DleXA5ADG78>

~~~
notatoad
I think it's pretty clear what's so interesting: google has the some of the
largest datasets ever assembled, on a hugely diverse range of subjects. If you
want to play with that data, you've got to work for google. Probably the only
other place you get to play with that much data is at the NSA, and their goals
are a bit less fun than google's.

~~~
simonster
There's also the fact that Google likely has more computing power than any
other organization worldwide, which is what seems to be the attraction for
Hinton, at least according to the last sentence of his post.

------
roquin
A related news is "Google acquires Canadian neural networks startup
DNNresearch, aims to improve image and voice search"

[http://thenextweb.com/google/2013/03/12/google-acquires-
cana...](http://thenextweb.com/google/2013/03/12/google-acquires-canadian-
neural-networks-startup-dnnresearch-aims-to-improve-image-and-voice-search/)

~~~
aflinik
not only related, that's exactly the same thing

------
oscargrouch
Google is sucking all the talent and the brain of the world.. i wish at least
that the efforts would get published in new papers and open source software..

Or the world will enter into a dark age of knowledge, ruled by huge
corporations, and everybody else having to share the crumbs that fall of the
table..

I hope that not all big minds fall of for companies.. or at least that
companies start to commit with knowledge shareing to the rest of the world

we should stop black box knowledge companies or we are doomed in the long
term.. (like the coke secret recipe case)

~~~
mayank
> i wish at least that the efforts would get published in new papers

You mean like this _very_ large list?
<http://research.google.com/pubs/papers.html>

> and open source software

Or this also very large list?
<https://code.google.com/hosting/search?q=label:google>

~~~
oscargrouch
Dont get me wrong.. google is much better than its "predecessors" in that
field.. eg: microsoft, apple, ibm.. but in the end is only one company, with
its capital open in the stock market, with boards and everything else.. while
google now is half good/half evil.. who can garantee that in the future when
larry and sergey goes away.. that the company wont be ruled by a Larry ellison
clone?

ps: while google have a pretty much decent portfolio of open source and papers
floating around.. its more about "marginal software" for google...

the core of its knowledge about the business still a secret.. what would be of
the world now and the cloud economy without Doug Cutting, that do it all by
himself?

In one sentence: is only ONE company rulled by the capital market and profits
and thats pretty scary.. thats all :s

~~~
kvb
I think this is unfair to at least Microsoft Research, which as far as I know
has outpublished Google by a wide margin. (Full disclosure - I'm a contractor
working on F#-related stuff at MSR)

~~~
ninjin
As I am not a contractor or affiliated with MSR in any way, I can back you up.
In my field MSR is light years ahead of Google when it comes to research
impact. Google has been gaining a stronger presence and they are generally
seen as more innovative. But for those of my colleagues that want an industry
research job, MSR and Google are both very attractive, the old stable one and
the new kid in town.

~~~
seanmcdirmid
(Disclaimer: I work for Microsoft Research Asia) On the other hand, what
Google does get around to publishing is often very strong. Outside of
universities, and since IBM lost/dropped/??? most of its researchers, we are
the only two big companies left that have strong ties to research.

------
tristanz
This is exciting for Hinton I'm sure, but also somewhat worrying for broader
community. Hinton has always been very open with his research:

[http://www.nytimes.com/2012/11/24/science/scientists-see-
adv...](http://www.nytimes.com/2012/11/24/science/scientists-see-advances-in-
deep-learning-a-part-of-artificial-intelligence.html?pagewanted=2)

One of the most striking aspects of the research led by Dr. Hinton is that it
has taken place largely without the patent restrictions and bitter infighting
over intellectual property that characterize high-technology fields.

“We decided early on not to make money out of this, but just to sort of spread
it to infect everybody,” he said. “These companies are terribly pleased with
this.”

~~~
simonster
I doubt Google will stop Hinton from publishing, and I think both will still
benefit greatly from that arrangement. Larry and Sergey published "The Anatomy
of a Large-Scale Hypertextual Web Search Engine" before they were even
incorporated, and it doesn't seem to have hurt them much.

------
jahewson
So is Google now more appealing to academics than academia?

~~~
ninjin
In a way, yes. Research in industry has the advantage that you don't have to
"waste" your time applying for grants in order to fund your own research
group. A lot of people seem to live in an illusion that professors actually
spend most of their time doing research, they are the managers of research
groups more than anything else and growing those research groups is hard work
and takes up close to all of your time (then add teaching to this). You are
unlikely to have more than a few permanent researchers under you and these
researchers will most likely be trying to leave within a few year and get a
professorship of their own.

Add to this that it is an enormous pain to solve engineering problems at a
university, you usually have to hand this to students, limiting what you can
produce and maintain. The best I have ever seen for a group was a single full-
time software engineer and this was at arguably the most prestigious
university in the world. And don't get me started on the fact that there is a
poor incentive to produce good software in academia, even though said software
can be essential to make research possible.

I don't know Hinton in person, but I can imagine that at his age both the
possibility of something new and the promise of a strong engineering and
financial backing for a large group is enormously tempting (also, do they
force professors into retirement or start denying them grants in Canada?). Oh,
and to those pointing out the potential salary, if my knowledge of the
financial situation of professors that have been on tenure is generalisable I
would be surprised if personal finances would mean much at this point.

~~~
drpgq
I won't put it up here, but those that are curious can look up Hinton's salary
at the University of Toronto by looking at the Ontario government sunshine
list website.

~~~
abrichr
Link for the lazy:
[http://www.fin.gov.on.ca/en/publications/salarydisclosure/20...](http://www.fin.gov.on.ca/en/publications/salarydisclosure/2012/univer12g.html)

------
kmfrk
Google is slowly becoming the equivalent of a Silicon Valley think tank. :P

Not that it's a bad thing. Lots of smart people did interesting things at IBM,
just as people at Microsoft Research are doing some great things - albeit
usually without anyone seeing the work in progress.

------
dchichkov
Poor grad students :-/

Anyway, if you are unfamiliar with his work, here's Hinton't Deep Learning
Saga ;) for your enjoyment: www.youtube.com/watch?v=mlXzufEk-2E

~~~
otoburb
Don't worry about them. He's still remaining attached to University of Toronto
part-time specifically because of his remaining graduate students[1].

[1] <https://plus.google.com/102889418997957626067/posts>

------
krmboya
Slightly off-topic, but I wonder what Dijkstra would feel about the state of
CS research if he came to the present time.

Hot topics now seem to be statistical ML stuff, and not the deterministic
mathematical proofs that he was an advocate of.

------
niggler
In contrast to many who think that people are being shepherded to the search
business, I wonder if Google is trying build a new "Bell Labs"-type research
division, in which case this is good news.

~~~
T-A
You mean something like <http://en.wikipedia.org/wiki/Google_X_Lab> ? :)

------
bronxbomber92
I read this as "Hilary Clinton" has joined Google, and did a big double-take!
Aha..

------
marcosploither
I have read George Clinton, it would have been a better headline.

