
Building high-level features using large scale unsupervised learning - marshallp
http://research.google.com/pubs/pub38115.html
======
forgotusername
Can we perhaps edit "singularity is near" out of the title? This sounds
impressive, but having a bunch of racks able to classify the outline of a face
is vastly disconnected from machine and humanity merging.

~~~
marshallp
It does this for 20,000 different objects categories - this is getting close
to matching human visual ability (and there are huge societal implications if
computer reach that standard).

This is the most powerful AI experiment yet conducted (publicly known).

~~~
tansey
> It does this for 20,000 different objects categories

With 15.8% accuracy.

> This is the most powerful AI experiment yet conducted (publicly known).

It's only powerful because they threw more cores at it than anyone else has
previously attempted. From a quick skimming of the paper, there does not
appear to be a lot of novel algorithmic contribution here. It's the same basic
autoencoder that Hinton proposed years ago. They just added in some speed ups
for many cores.

It's a great experiment though. You shouldn't detract from its legitimate
contributions by making outlandish claims.

~~~
redmoskito
To a computer vision researcher, 15.8% on 20k categories is phenomenal.

------
nl
People are missing the point here.

Yes, 15% accuracy doesn't seem great.

BUT the detector built its own categories(!). It managed to find 20,000
different categories of objects in Youtube videos, and one of these categories
corresponded to human faces, and another to cats.

Once the experimenters found the "face detection neuron" and used it to test
faces THAT neuron managed 81.7% detection rate(!).

Forget the singularity, and just think about how amazing that is. The system
trained itself - without human labelling - to distinguish human faces
correctly over 80% of the time.

~~~
agravier
> BUT the detector built its own categories(!).

It's not revolutionary. Clustering algorithms and neural nets are plenty.

Really, what differentiates this network is its scale.

~~~
abrahamsen
And, thanks to scale, that some of the clusters corresponds to high level
concepts. According to the article, earlier attempts have mostly resulted in
low level concepts like "edge" or "blob" to be detected.

Also, it was (again, from the article) plausible but not a given that high
level concepts could be found from unlabeled data.

That "cat" is one of the high level concept you get from using random Youtube
videos as raw data is both impressive, and slightly amusing.

~~~
agravier
Exactly. Regarding your remark about edge detectors: such self-organizing
neural nets are organized into hierarchical layers, and early layers' units
are going to learn to become detectors of statistically common components of
the input image, in the same way as the initial layers of the visual system
perform blob and edge detection (retina, lateral geniculate nucleus, V1). In
mathematical terms, these early units learn the conditional principal
components of the inputs. The layers that are built upon these detectors, if
correctly organized, are going to build upon this initial abstraction and
learn more complex features: for instance to find these these edges in
relative positions (to each other). Eventually, up the abstraction chain,
units detect such statistically frequent features as the shape of cat's ears
(common in youtube videos, I imagine), etc...

(Sorry I wrote that fast, I hope it's understandable)

------
Homunculiheaded
There was an interesting discussion on Quora about this recently[0]

The most relevant quote being perhaps:

"The magic of the brain is not the number of neurons, but how the circuits are
wired and how they function dynamically. If you put 1 billion transistors
together, you don't get a functioning CPU. And if you put 100 billion neurons
together, you don't get an intelligent brain."

0\. [http://www.quora.com/How-big-is-the-largest-feedforward-
neur...](http://www.quora.com/How-big-is-the-largest-feedforward-neural-
network-ever-trained-and-what-for)

~~~
eric_bullington
That's an interesting discussion, but this experiment suggests exactly the
opposite (perhaps that's why you included the discussion). Who knows, if we
put 1 billion cores together, and fed it a massive amount of data (akin to
what a baby receives as he/she matures), perhaps we would get a brain we would
consider "intelligent". The fact that this system was able to pick out high-
level features like "face" and "cat" without any prior training -- and with
only 1000 cores, not 1 billion -- is quite suggestive that they're on to
something.

EDIT: Mistyped number of cores. 1000, not 100.

~~~
JohnHaugeland
"That's an interesting discussion, but this experiment suggests exactly the
opposite (perhaps that's why you included the discussion)."

It absolutely does not. This experiment supports that position strongly.

What this experiment shows is that said meaningful structure can be
progressively, automatically discovered.

"and with only 1000 cores, not 1 billion"

Comparing CPU cores to individual neurons is more than slightly disingenuous.

------
seiji
16,000 cores sounds impressive until you realize it's just five to ten modern
GPUs. For Google, it's easier to just run a 1,000 machine job than requisition
some GPUs.

See: <http://www.nvidia.com/object/tesla-servers.html> (4.5 teraflops in _one
card_ )

Reminder: GPUs will destroy the world.

~~~
Symmetry
What a GPU calls a "core" doesn't at all correspond to what a CPU calls a
"core". Going by the CPU definition (something like "something that can
schedule memory accesses") a high end GPU will only have 60 or so cores. And
going by the the GPU definition (An execution unit) a high end CPU will tend
to have 30-something cores.

GPUs do have fundamentally more execution resources, but that comes at a price
and not every algorithm will be capable of running faster on a GPU than on a
CPU. If neural networks just involve multiplying lots of matrices together
with little branching they might be well suited to GPUs, but most AI code
isn't like that.

~~~
JohnHaugeland
"What a GPU calls a "core" doesn't at all correspond to what a CPU calls a
"core"."

They aren't as different as you imagine. They're general purpose programmable
arithmetic units with processing rates on the order of 20-30% of CPUs,
provided the limitation that they're all doing roughly the same thing.

For most machine learning tasks, that's exactly what you're doing anyway. Oh
no, your neural network engine has to be parallel? C'est damage!

~~~
Symmetry
A CPU core isn't a general purpose programmable arithmetic unit, though. In
fact what you call a "core" when you're talking about CPUs is composed of
_multiple_ such general purpose programmable, as well as less general purpose
memory load/store units that can still be used for basic arithmetic and a
instruction fetch and scheduling system. So a core in you Intel iFOO processor
is structurally equivalent to what NVidia calls an SM. Now, and NVidia SM has
48 execution units to Intels 6, but it operates at a lower frequency and
doesn't have the bypass network, branch predictor, memory prefetcher, etc that
you could find in an Intel core. So there are some tasks where the Intel core
will be much faster than the NVidia SM, and some tasks where the NVidia SM
will be much faster. And the case here does seem like one where the GPU has an
advantage. But saying that the NVidia GPU has 1526 "cores" is just dishonest.

~~~
JohnHaugeland
"In fact what you call a "core" when you're talking about CPUs is composed of
multiple such general purpose programmable, as well as less general purpose
memory load/store units"

So are GPU cores.

"But saying that the NVidia GPU has 1526 "cores" is just dishonest."

No, it isn't. You can run 1536 things in parallel at speeds that would have
qualified as full cpu speeds several years prior.

Something isn't any less a core merely because it does less juggling magic,
and that juggling magic is actually undesirable for a heavily parallelized
task.

"So there are some tasks where the Intel core will be much faster than the
NVidia SM, and some tasks where the NVidia SM will be much faster."

This conversation already has a context. Arguments which ignore that context
completely miss the point.

If you don't understand how I achieved the amount of processing I did, that's
fine. Playing games with the semantics of a "core" somehow magically requiring
all the features of current Intel-strategy chips, though, are not going to
convince me.

There is more to Heaven and Earth, Horatio, than is dreamt of in Intel's
philosophy. This sort of attitude towards what constitutes the no true
scotsman "a real core" is why Arm is in the process of eating Intel alive, and
why Tilera stands a decent chance of doing the same thing to ARM.

This is merely extreme RISC. I realize it's sort of a tradition for the modern
VLIW movement to suggest that if you can't double-backflip through a flaming
hoop made out of predictive NAND gates it somehow doesn't count.

But, if you actually look, the rate of modern supercomputing going to video
cards is rising dramatically.

So obviously they count as cores to _somebody_.

You also seem to have missed the point. It's not the core scale that we're
discussing here. It's the dataset scale. The number of cores you throw at a
problem is not terribly important; 20 years ago it would have been
breathtaking to throw 32 cores at a problem, and now that's two CPUs.

What makes an experiment cutting edge is the nature of the experiment, not the
volume of hardware that you throw at it. I was talking about the /data/ and
the /problem/ . Predicting movie ratings is a hell of a lot harder than
feature detection.

------
JohnHaugeland
The singularity is a poorly constructed myth. It is built around the
presumption that intelligence is a linear function of CPU power, and that
surely as CPU power rises, so shall intelligence; the problem is, that
prediction was made in the 1970s, since which CPU power has risen ten decimal
orders of magnitude, and we still don't have much better speech recognition
than we did back then, let alone anything even approaching simple reasoning.

The ability to detect faces is not a signal that general intelligence is right
around the corner.

~~~
Symmetry
Well, maybe. There are a whole lot of very different things called "The
Singularity" and some of them are much more reasonable than others.

There's the Cambpellian Singularity, which says that we won't be able to
predict what will happen next. Pretty non-controversial as far as it goes.

There's the Vingean Singularity, which says that if we ever develop AIs that
can think as fast and as well as humans then due to Moore's Law they'll be
thinking twice as fast as humans after 2 years, so they'll start designing
chips and the period of Moore's law will fall to 1 year, and so on with us
reaching infinite computing power in finite time. I think this vision is
flawed.

Relatedly, there's the Intelligence Explosion Singularity (associated with
Yudkowsky), which says that as soon as its AIs designing AIs, smarter AIs will
relativly quickly be able to make even smarter AIs and we'll get a "fwoosh"
effect, though not to infinity in finite time. I find this unlikely, but can't
rule it out.

There's one I don't have a handy name for, but lets call it the AI Revolution
viewpoint, which is that AIs will cause civilization to switch to a faster
mode of progress, just like the Agricultural Revolution and Industrial
Revolution did. This one will only look like a singularity in hindsight, and
might seem gradual to the people living through it. I think this one is pretty
credible.

There's the Kurzweilian Singularity, where thanks to Accelerating Change we'll
someday pass a point which will arbitrarily be called the Singularity. As far
as I can tell this is Kurzweil appropriating the hot word of the moment for
his ideas a la Javascript.

Then there's the Naive Singularity, which equates processing power with
intelligence and then concludes that computers must be getting smarter. This
is indeed totally naive and not something we should worry about. I guess the
linked paper is evidence that you can substitute a faster computer for smarter
AI researchers to some extent, but probably not a very large one.

~~~
JohnHaugeland
"There are a whole lot of very different things called "The Singularity" "

If the singularity was a legitimate concept with anything approaching
experimental evidence, then this could not be true. This observation of yours
- with which I agree - suggests to me that The Singularity needs a pope hat.

It is instructive to notice that all of "the singularities" are the products
of science fiction authors, and in the case of the original, a particularly
bad one.

There is a delightful level of schadenfreude involved in observing the
multiplicity of "The Singularities." In two different ways its name says
"there's only one," and yet they still can't agree on topics that are critical
and fundamental to the concept itself, like the definition of intelligence, or
whether or not to circumcise.

Pass the sacramental chalice, please?

~~~
Dn_Ab
Vinge is a retired math professor, I.J. Good was an accomplished
mathematician, Kurzweil makes hard to swallow predictions but is still an
accomplished technologist and I happen to very much enjoy Vinge's writing.
Regardless, the idea is worth considering independent of who is saying it.

I am also sure you know that words - such as variety and polymorphism - have
different context specific meanings. Singularity in this case as in the kind
of thing you can find on a variety but not on a manifold.

The idea of infinite recursive Moore's law fueled intelligence explosions
leading to super human intellects by 2030 is something I assign a low
probability to. I don't find it hard to believe that there is some point in
the future - say 2131 - such that if anyone alive today or previously were
transported there, they would never be able to understand what was going on
and _everyone_ from that time would think circles around them.

~~~
JohnHaugeland
Maybe you didn't realize this, but the science fiction author bit wasn't meant
as a slander. Many high quality people, and also Kurtzweil, are science
fiction authors.

What I was getting at was "you realize they're writing books to make people
happy for money, not doing legitimate science on that day, right?"

.

"words - such as variety and polymorphism - have different context specific
meanings."

Sure. All handwaving about the rules of language notwithstanding, though, none
of The Singularities have merit or underlying measurement, even if you want to
talk syntax and grammar to create a seeming of academia by proxy.

.

"The idea of infinite recursive Moore's law"

... is nonsense. What would "recursion" be in the context of Moore's law? Have
you even thought this over?

What, Moore's Law solves itself by going deeper into itself until the
datastructure is exhausted?

.

"fueled intelligence explosions"

The science fiction part. I mean, you might as well say "fuelled by warp
drives," because there's no evidence they're going to happen either. Or
unicorns.

.

"is something I assign a low probability to."

This suggests that you don't know what probabilities are. Probabilities are
either frequentist, which cannot happen here because we have no knowledge of
the rates here (this would be like calculating the frequentist probability of
alien life - it's just making numbers up,) or Bayesian, where you draw
probabilities from observed events, at which point the probability is exactly
zero.

So, is it undefined or zero that you're promoting?

.

"I don't find it hard to believe that there is some point in the future - say
2131"

(rolls eyes)

.

"they would never be able to understand what was going on and everyone from
that time would think circles around them."

It seems you don't even need to be transported into the future for that.

~~~
Dn_Ab
1.) Okay.

2.) Singularity as in breakdown not as in single. You purposely muddled the
meaning to make your quip work.

3.) Moore's law fueled as in AI gets interest on their intelligence. Recursive
as in AI makes smarter AI makes smarter AI...

4.) Science fiction or not i find it unlikely.

5.) Bayesian. Look up prior.

6.) 2131 was tongue in cheek.

7.) Thanks ;) You actually never address my main point though.

~~~
JohnHaugeland
"2.) Singularity as in breakdown not as in single. You purposely muddled the
meaning to make your quip work."

This is a blatant falsehood. I have solely and exclusively used it as a title
for Kurtzweil's concept. It has no meaning; it's a name. I have muddled
nothing. It is inappropriate for you to make accusations like this without
evidence.

.

"3.) Moore's law fueled as in AI gets interest on their intelligence"

Yes, that's what I said at the outset: this whole thing is driven by the false
belief that intelligence is a function of CPU time. There is no experimental
evidence in history to support this, and there are 65 years of counter-
examples.

Repeating it won't make it less wrong.

.

"Recursive as in AI makes smarter AI makes smarter AI..."

Oh.

This gets to a different false presumption, namely that the ability to create
an intelligence, as well as that the power of the intelligence created, is a
linear function of the prior intelligence.

This whole treating everything like it's a score, like it's a number you tweak
upwards? It's crap.

You can't make an AI with an IQ of 106 just because you have a 104, and the
guy who made the 104 had a 102.

This is numerology, not computer science.

.

"4.) Science fiction or not i find it unlikely."

I can't even tell what noun you're attached to, at this point. What do you
find unlikely?

.

"5.) Bayesian. Look up prior."

What about bayesian, sir? I don't need to look up prior; I used it, correctly,
in what I said to you. You're just telling me to look things up to pretend
that there is an error there, so that you can take the position of being
correct without actually having done the work.

There are zero priors of alien life, sir. That was my point, in bringing up
what you're now blandly one-word repeating at me, in your effort to gin up a
falsehood where none actually exists.

.

"7.) Thanks ;) You actually never address my main point though."

You don't appear to have one.

Maybe you've forgotten that you were replying to someone else, who already
said that to you?

------
chmike
The only originality in this work is the processing power used. The principle
in it self is not original.

For a possible resemblance with the real cortical neural network working
principle and face or object recognition, this is just a farce.

Regarding getting closer to the presumed singularity, this is like saying that
cutting flint is close to making diamonds.

The authors didn't claim that, but the abusive use of "neural network" for
such kinds of applications is just doing that. It is a dishonest abuse of
people who can't make the difference.

The true problem is that significant quality work toward modeling real
cortical neural network is drown in the sea of such faker crap.

~~~
_delirium
I think much of this is overhyped as well, but I disagree that whether
modeling human brain structure is relevant. The term "neural network" has
historical baggage (responsible for some of the hype), but these days refers
to a class of mathematical approaches with only historical connection to
"neurons". Those can be interesting on their own for AI purposes, and imo
accurate modeling of the human brian, while interesting for neuroscience
research, is not necessarily the way forward for AI research.

~~~
chmike
My use of modeling was indeed inappropriate. What I meant is the working
principle of cortical neural networks. The artificial model would then be a
kind of proof of concept.

Regarding you other point, it is a matter of research strategy. I think that
the path trying understanding the working principle of real cortical neural
network is the shortest path to AI. My impression is that the other path which
is to play around with artificial neural networks is too hazardous.

We can make a parallel with learning to fly. We are in a similar situation
regarding how the brain works and AI. Understanding how birds fly require a
true research. People seems to simply focus on flapping while this is not the
real working principle of flight.

I see there a strong analogy with artificial neural network. The most relevant
properties of cortical neural networks are ignored.

With flight the proof condition of mastering it was obvious. With AI, it is
less obvious. I would be glad to hear suggestions. Face recognition is the
most difficult condition because this process is the end product of many prior
processes like 3D perception and feature extractions. My current impression is
that talk decoding would be a much better candidate. Siri shows the potential
impact of such AI product. At least the turing test would be a direct match.

~~~
_delirium
I guess I take the opposite suggestion from the flight example: we succeeded
in flying once we started focusing more on the physical/mathematical research
of aerodynamics and lift, and less on attempting to mimic biology, in copying
the biomechanics of bird wings. We ended up producing something that flies,
but not in exactly the way that birds fly. That's what I tend to view as the
better route for AI as well: instead of trying to copy the details of how a
brain works, focus more on first-principles mathematical/logical principles of
inference, whether they're symbolic ones (e.g. theorem-proving) or statistical
ones (Bayesian networks, etc.).

Admittedly this is a big area of disagreement both within and outside the
field.

------
jamesaguilar
You'd be surprised at how little precision this much computing power gets you,
even on very basic classifiers. If anything, working on this kind of stuff has
given me an appreciation for just how far away the singularity may be.

------
ekianjo
15.8% accuracy != singularity is near.

No more supplements-eating Kurzweil, walking Terminators and Skynet-like BS
please.

~~~
scotty79
I wonder what accuracy would human get if you trained him/her only with 10
million static 200x200 px images in complete silence.

~~~
ekianjo
I don't think this was "complete silence" in this case. They actually trained
the computer (meaning they provided data as input).

~~~
ehsanu1
scotty79 means to showing a human _only_ the 10 million 200x200 images. We
probably see a lot more than that, in 3D from different perspectives with
continuity in motion, and in a lot more detail and possibilities for
filtering.

~~~
ekianjo
> scotty79 means to showing a human only the 10 million 200x200 images

Then, what kind of thing are you measuring? Recognizing patterns ? We know
humans are very good at that. We can see thousands of different people
everyday but we can recognize in the blink of an eye a familiar face. A
computer or computer network is very, very, very far from being able to do
that yet.

~~~
kalleboo
But why are humans are very good at that? Could it be because humans have
gotten tons of high-resolution, 3D imagery with binaural audio? Whereas this
computer got a handful of low-res still images. Is the solution to just throw
more horsepower at the problem, or is there really some inherent quality of
the brain that's different?

edit: this quote puts things into perspective a bit "It is worth noting that
our network is still tiny compared to the human visual cortex, which is
1,000,000 times larger in terms of the number of neurons and synapses."

~~~
ehsanu1
Also, consider training time. I doubt babies are recognizing 20k objects at
15% success rate after just a few days. Though to really compare that, the
speed of the brain vs the speed of the supercomputer has to be normalized for,
and training data and network size has to be similar as well of course for any
_real_ comparison.

------
jpeterson
Maybe I'm missing something here, but how exactly is it "unlabeled" data if
they're specifically feeding it millions of pictures of "faces"? I mean, if
you make a specific selection of the type of images you train the network on,
isn't that basically equivalent to labeling them?

~~~
Smerity
The aim of the paper was to produce an unsupervised system that would generate
high level features from noisy data. These high level features could then be
used in supervised systems where labelled data is added.

Thus, the paper is about using an unsupervised system to help a later
supervised system. An advantage of this is that, as the unsupervised system
isn't trained to recognise object X, it instead learns features that are
discriminative. This same network could be used to recognise arbitrary objects
(which is what they do later on in the paper with ImageNet).

~~~
leot
In other words: imagine a baby. She sees 100k "images" of faces. Thanks to the
statistical regularity of the world, she now has a "subsystem" that recognizes
a face _in the absence of her knowing what it's called_. Then, when she is
told "this is a face" she pins this thing pointed-to to the existing, unnamed
representation.

------
kenrikm
While I'm excited about progress, 15.8% accuracy is not exactly "Singularity
is near"

~~~
marshallp
They just have to scale it up - more computers, more days, and the accuracy
level should increase accordingly (that is my intuition and hope on this,
though i could be wrong).

~~~
heretohelp
Haha, no.

\--- last company was in computer vision.

~~~
roel_v
"last company was in computer vision."

Did you sell, leave or did it fail? Why? I have some ideas that I think are
novel applications of computer vision, and just within the range of what's
feasible, but it seems that most computer vision applications look like that
at first, and then after 90% done find out that the second 90% is
exponentially harder and, realistically, infeasible. How could I test my ideas
against that? Or am I asking from wrong premises?

~~~
heretohelp
Company was doing well and had a good idea. Product worked great, we did our
job. The problem is, management didn't.

I left after all the other engineers did.

------
semisight
This isn't singularity material. While this may not be a bog standard neural
network, it has no feedback. It cannot think, because thinking requires
reflection. It is trained by adjusting the weights of the connections after
the fact using an equation.

Is it cool, and perhaps even useful? Yes. But don't confuse this research
project for a precursor to skynet.

~~~
marshallp
I put the singularity bit in to make it relevant to people who would otherwise
not get the significance of this (which is that large scale neural nets can
work - something people have been trying and failing at for decades).

~~~
Chlorus
Or you put it in as a cheap link-bait tactic.

~~~
marshallp
The site is google research - I have no ads on it. This paper's been out for
weeks but no mentions anywhere - thought i would give it a deserving push.

------
cbhl
"We also ﬁnd that the same network is sensitive to other high-level concepts
such as cat faces..."

"Our training dataset is constructed by sampling frames from 10 million
YouTube videos."

------
karpathy
If anyone is interested to read more on this topic, there is another recent,
closely related and perhaps slightly more accessible paper ( "High-Level
Invariant Features with Scalable Clustering Algorithms" <http://bit.ly/KDuN04>
) from Stanford that also learns face neurons from unsupervised collection of
images (disclaimer: I'm co-author). It uses a slightly different model based
on layers of k-means clustering and linking, but the computation in the end is
very similar.

I'm familiar with both models so I can also try to answer any questions.

~~~
bfrs
I think your comment <http://news.ycombinator.com/item?id=3838971> is very
relevant to this discussion.

------
sown
I'm seriously considering quitting my job and studying ML for a few months in
a desperate attempt to get work in projects like this. I feel like I'm missing
out but too dumb for traditional grad school.

~~~
p1esk
That's what I did a few months ago - quit my job and decided to go to a grad
school to study AI (with focus on neural nets and ML).

~~~
ralfd
What did you learn so far about neural nets? I recently looked into machine
learning and naively thought I could find at least one practical fun tutorial
"Here is a neural network API in C, you have to do that and this to let a
simulated robot evade obstacles or learn to play Asteroids". Instead my
(extremely superficial) search did find that neural nets are pretty arcane,
genetic algorithms trapped in local minima and you are faster and better of
coding logic yourself, developing a mathematical model to calculate results,
instead searching for patterns in vast sets of data.

~~~
caster_cp
SVMs are a relatively easy to use (but not to understand) method that yield
impressive results for beginners. See the libsvm website, there are plenty of
good material there (<http://www.csie.ntu.edu.tw/~cjlin/libsvm/>). But as
stated somewhere else here, game AI is a whole different story.

------
epaga
Fascinating, exciting times we live in. I'll upvote the link as soon as the
"singularity" silliness is taken out of the title.

------
dchichkov
From the article: "It is worth noting that our network is still tiny compared
to the human visual cortex, which is 1,000,000 times larger in terms of the
number of neurons and synapses."

------
DanBC
> _the dataset has 10 million 200x200 pixel images downloaded from the
> Internet_

They take the frames from YouTube. It is weird to me that YouTube, (derided as
a way of sharing funny cat videos) is able to contribute something actually
useful to the world.

~~~
manmal
Youtube contains a lot of educational material you would not find otherwise.
If I wanted to learn sewing on a sewing machine, I would just watch some video
tutorials - try that with a book on sewing. Same thing with instructions on
how to play instruments. Heck you can even watch videos on how to fix problems
with your car engine. Many procedural instructions can't be transported
properly via papers or books. It also allows asynchronous video messaging for
laymen asking experts stuff which is difficult to get via written text. I bet
that youtube will contribute very much to knowledge preservation and
distribution in the long term.

~~~
DanBC
I hope that Google look after it a bit better than they've looked after their
Usenet archives. Google groups search is a particularly frustrating
experience.

I agree that there is some great content on Youtube. Interesting that you
mention sewing machines, because that's something I've used and they are
particularly helpful. (See also all those other crafting videos; latch-hooking
etc.)

------
mbq
As far as I can tell, this is let's train a huge number of models and then
cherry-pick few that works well on a test set, so an overfitted junk. What
have I missed?

~~~
moultano
It seems like they are cherry picking the one that works well on the training
set no? Where did you get the sense that they were doing it on test?

~~~
mbq
The train is not labeled, so it is not possible; and they do not mention that
the labeled set was split or used in validation -- it is just called "test".

~~~
moultano
"We followed the experimental protocols speciﬁed by (Deng et al., 2010;
Sanchez & Perronnin, 2011), in which, the datasets are randomly split into two
halves for training and validation. We report the performance on the
validation set and compare against state-of-theart baselines in Table 2. Note
that the splits are not identical to previous work but validation set
performances vary slightly across diﬀerent splits."

~~~
mbq
As I understand, this is only about this side experiment with ImageNet data
which uses logistic regression on those neurons in some cryptic way; I was
trying to comprehend the core work (faces) before that.

~~~
moultano
Well, that's the record breaking bit.

------
RyanMcGreal
We need to harness this neural net to improve kittydar:

<https://news.ycombinator.com/item?id=4116990>

------
panacea
The singularity is already here. There are black holes in our universe. This
submission title really annoys me.

~~~
MindTwister
Even if you are trolling, I think I'll leave this here.
<http://en.wikipedia.org/wiki/Technological_singularity>

~~~
panacea
I'm not trolling, I've read Keurzweil's book... It's extrapolating. If I was a
risk assesor, I'd err on the side of a mass extinction event over us consuming
the entire universe as data.

~~~
dmix
There's already been 4 extinction events, it's pretty safe to bet that it will
happen again.

But they happen 50-100 million years between each other and usually take
thousands of years to take full effect once they begin.

Even if technological singularity takes an extra 100-200 years to really
happen, if any significant 'AI' is achieved, a lot could happen in a thousand
years, let alone a million.

~~~
JohnHaugeland
For what it's worth, your ability to post that is based on that they were
actually near-extinction events.

------
tensor
Is accuracy referring to recall, precision, or some other measure?

~~~
andreasvc
When there is no possible difference between recall & precision, you report
one figure, accuracy.

------
marshallp
Google Research and Stanford researchers test out a 1 billion connection, 9
layer, 16,000 core deep learning neural network (see geoff hinton and andrew
ng talks on youtube) to recognize 20,000 different objects in images (with low
accuracy but huge improvement over previous approaches).

This was done no with no pre-labeled images (except for fine tuning)! A brain
that learned from raw images. The same algorithm can be applied to any data
type (financial data, text, audio, images/video) without any human involvement
(except gatheting of unlabelled data and running the system). Pretty much the
artificial intelligence holy grail!

~~~
Cyranix
"Singularity is near"? "Holy grail"? You may be getting a little carried away
here.

The outcome shows a very nice improvement on an unsupervised classification
and feature detection task, but it also highlights that unsupervised machine
learning still has a long way to go. 16% accuracy from a network with 1bn
connections and 100m inputs using (if my math is right) 1.15m hours of CPU
time. Which of these would be the easiest way to continue making gains:
investing more time/hardware, increasing the complexity of the model, or
developing a new and improved algorithm altogether? All of these sound pretty
intensive to me.

~~~
marshallp
If the algorithm keeps increasing in accuracy as you scale up computation and
add more unlabeled data that is pretty amazing. You might get something that
matches human performance on vision/speech recognition etc.

~~~
jules
If you extrapolate that way you'd conclude that naive Bayes is the solution to
AI. Improvements tend to tail off fairly quickly as you add more data and
computation, unfortunately.

------
adventureful
The added singularity is near part is silly. In ten years when that's a
500,000 (or 5 million) core neural net, the singularity is near part still
won't apply.

------
mjwalshe
I thought that the best so far was simulating 1/2 a mouse brain compextity at
1/2 speed.

~~~
SafeSituation
Where can I find this study?

~~~
ehsanu1
He might be referring to blue brain, though I recall that simulated 100k
neurons from the neocortex (not quite half a mouse brain). The thing is that
they simulated the physical processes in the synapses and more for a more
accurate representation, so it's actually a lot cooler than it sounds!

Here's a link I googled up: <http://bluebrain.epfl.ch/cms/lang/en/pid/56882>

------
PaulHoule
"we trained our network to obtain 15.8% accuracy"

Film at 11

~~~
snikolov
Look at this way: for a 20000 category classification problem, guessing
randomly will give you 0.005% accuracy. Compared to chance, this is pretty
good.

