

What I learned from competing against a ConvNet on ImageNet - Houshalter
https://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/

======
dave_sullivan
_Based on the sample of images I worked on, the GoogLeNet classification error
turned out to be 6.8% (the error on the full test set of 100,000 images is
6.7%). My own error in the end turned out to be 5.1%, approximately 1.7%
better._

To put this in perspective, the task is basically "given a picture, identify
the correct class out of 1000 categories" (it gets into specific breeds of
dog, for instance). Humans turn out to miss some of these, and neural networks
are now nearing human level performance.

 _It 's fun to note that about 4 years ago I performed a similar (but much
quicker and less detailed) human classification accuracy analysis on CIFAR-10.
This was back when the state of the art was at 77% by Adam Coates, and my own
accuracy turned out to be 94%. I think the best ConvNets now get about 92%._

Things are moving quickly in this field.

 _(My personal) ILSVRC 2014 TLDR: 50% more teams. 50% improved classification
and detection. ConvNet ensembles all over the place. Google team wins._

------
bainsfather
To be clear - this is a _human_ viewing the images, and only narrowly beating
the neural net.

Furthermore, the human spent some time 'training' himself on the images, plus
he used a neural net to reduce the possible answer space - he didn't come to
it cold.

"Then I organized a labeling party of intense labeling effort only among the
(expert labelers) in our lab. Then I developed a modified interface that used
GoogLeNet predictions to prune the number of categories from 1000 to only
about 100. It was still too hard - people kept missing categories and getting
up to ranges of 13-15% error rates. In the end I realized that to get anywhere
competitively close to GoogLeNet, it was most efficient if I sat down and went
through the painfully long training process and the subsequent careful
annotation process myself."

------
brotchie
Go to the demonstration labeling interface here:
[http://cs.stanford.edu/people/karpathy/ilsvrc/](http://cs.stanford.edu/people/karpathy/ilsvrc/)

Hit the "Use hard course".

On the left hit "Show google prediction" consider the answers for a few
seconds, then hit "Show answer".

My reaction to the Google predictions is "yeah, that's reasonable". Often my
reaction to the actual true label is "wtf, that's not obvious at all."

I get the feeling the Google image net produces better tags than the
validation set labels :)

------
mdda
There are some good TLDRs here. But the actual article includes a link to a UI
with the actual ImageNet decision task created by the author. You can compete
against the winning ConvNet too..

------
ap22213
This is very impressive. I hadn't realized that image classification was so
good.

I'm not sure I understand the results though. On the demonstration site,
there's a group of images that apparently failed to be classified. But, when I
tried it this morning, google's prediction shows a correct classification, for
some of them.

Anyway, where is this service? It would be extremely valuable to developers,
even in it's current state.

