
Google's Street View computer vision can beat reCAPTCHA with 99% accuracy - apawloski
http://googleonlinesecurity.blogspot.com/2014/04/street-view-and-recaptcha-technology.html
======
zwegner
This particular issue (AI performance on captchas) is really quite
fascinating. It's an arms race, but the problem is, only one side can win.
Google is claiming they have improved their system in some (understandably)
unspecified way, but there's only so far this can go. Captchas need to detect
whether someone is human, but it has to work for everyone, ideally, even those
with disabilities. Any simple task a human can do will eventually be able to
be automated. Tasks that aren't currently feasible to be automated, say some
natural language processing tasks, have another problem: scalability. To
prevent simple databases of problems -> solutions, the problems need to be
generated en masse, and for cheap, which means a computer needs to generate
the solutions in the first place. And of course, paying people to just do
captchas all day already happens.

The street address/book scan approach that Google uses is interesting in that
the exact solution is not known, so they presumably have to be somewhat
forgiving in accepting answers (as their machine learning might have gotten it
wrong). Perhaps this is what their "risk analysis" refers to--whether their
response seems "human" enough according to their data, not necessarily whether
it's correct.

I don't see a way around this problem for free services that still preserves
privacy (so directly using some government-issued ID is off the table). Maybe
some Persona-like digital signature system, where a person can go to a
physical location with a government ID, and get a signature that says "Trusted
authority X affirms that person Y is in fact a person". Obviously this still
has problems, as you need to trust X, and it's a big pain in the ass.

There are parallels to the realm of passwords, which are also becoming
obsolete (not that there's a good replacement...). Anything that a human can
feasibly remember for a bunch of sites is becoming easier and easier for
computers to guess.

So basically, computers are taking over the world, and we can't do anything to
stop it. God help us all.

~~~
daenz
> It's an arms race, but the problem is, only one side can win.

I don't disagree, but although this specific problem can be automated, the
arms race will continue as newer problems (requiring uniquely human
intelligence) are presented. We can keep moving the bar until the singularity.

For example, a captcha: "Pick the most unsafe environment", followed by
pictures of a house, a park, a volcano, a bed. To solve this, AI would have to
have a mastery of language, a mastery of object identification, and a mastery
of the metadata of those objects as they apply to human safety.

~~~
zwegner
That runs into the scalability problem that I mentioned--someone (say, a
Mechanical Turk user) can solve that problem once, and put it in a database.
To avoid this, there needs to be a way to generate them cheaply, so you pretty
much need a computer that can solve the problem in the first place.

Basically, my point is that there's no such thing as "uniquely human
intelligence". There's lots of problems that fall into that category _now_ ,
but with the continuing rise of computational power, it's getting harder and
harder to find new problems of this sort without excluding a lot of real
people.

~~~
adw
So that begins to look a bit like the RSA problem, doesn't it? Find a problem
which can only be solved efficiently given some "secret" knowledge – where,
instead of that knowledge (the prime factors) being secret, the knowledge is
something it's easy for a computer to _use_ but hard for it to _recognize_.

I'm probably overgeneralizing, but it's an interesting framing.

------
josho
Interestingly I activated a new gmail account today and during the signup
process I experienced the obligatory captcha. It was in two parts, the first
looked strikingly like a street view picture of a house number, while the
second looked like a traditional captcha.

I suspect that google has been using techniques like this to validate their
computer vision conclusions. Which makes their 99% assertion even more
interesting, because it's likely 99% confirmed by a very large crowd sourced
data set, not simply a staff member going through several hundred samples to
come up with the success rate.

~~~
id
It has been like this for a while now. AFAIK you have either

    
    
      - the actual captcha and a word from a book (Google Books)
      - the actual captcha and a house number from Street View (Google Maps)
    

Only the actual captcha has to be typed in correctly.

~~~
ashmud
Do people ever intentionally type in nonsense for the non-captcha part, out of
spite, for instance? If you could persuade enough people to provide non-
answers for the non-captcha part, would Google stop using it (the non-captcha
part)? My guess is it's like spam. As long as it works a small fraction of the
time, it will still exist.

~~~
akgerber
I did when Recaptcha came out and it was consistently obvious which was the
unknown part. It's no longer so obvious, in my experience.

I believe ReCaptcha presents the same unknown value to several users, and
rejects outliers.

~~~
id
In my experience, it's still extremely obvious. It's always the word that's
less readable (and doesn't look like it came from a book). And if you have
numbers, it's even easier.

Recently they added special captchas for untrusted users. If they think you
might be a bot, you'll get much harder captchas.

[http://googleonlinesecurity.blogspot.com/2013/10/recaptcha-j...](http://googleonlinesecurity.blogspot.com/2013/10/recaptcha-
just-got-easier-but-only-if.html)

~~~
path411
Actually often it can be the opposite of what you would think. I don't think
Google will distort the "unknown" word as much as the "known" word, making it
where the "known" word can be harder to read for a human because it's passed
through more filters.

------
adyus
In effect, Google computer vision got so good that they made their own system
obsolete. This is a good thing.

I still think the only reliable way to confirm identity (or humanity) online
is an email or SMS verification. Recently, receiving a 2-factor SMS code took
less time than the page refresh prompting me to enter it.

~~~
astrange
That confirms identity, but it doesn't confirm humanity. Robots just need to
have a SIM card!

I think it would annoy many human users as well.

~~~
snewman
It makes it much harder to flood a given site with a large number of actions.
For instance, if you require SMS verification for account signup, and you
don't allow the same phone number to be used for more than, say, two signups
per month, then you make it hard to create a large number of bogus accounts.

Yes, it will annoy some real users, so it's not a no-brainer. But it will put
a major brake on many kinds of abuse.

~~~
wpietri
Your underlying hypothesis here is that phone numbers are hard to get. But
Twilio only charges $1/month per number, and presumably gets them for a lot
less than that. If you use them for multiple nefarious purposes at once, your
cost-per-evil could be pretty low.

And that's assuming that you just don't steal them outright. Suppose you boost
somebody's Android game, repackage it, and add a little code that intercepts
and replies to certain text messages. It'd basically be a SMS botnet.

SMS verification would make certain kinds of casual abuse harder, but I don't
think it'd be a big barrier for well-organized assholes. E.g., the "Rachel
from Cardholder Services" people that have been using the murkiness of the
telephone network to sneak billions of illegal calls past the FTC.

------
dnlbyl
99% is probably better than my success rate with reCAPTCHA...

~~~
yaddayadda
Looking at the four examples they give in the article, I'd likely only get one
or two on the first go.

\- ntrtyLt (I'm mostly confident)

\- tmincrw (I'm really confident)

\- rrnmtht (I'm not at all confident of my answer, but I'm really confident
that r's, n's, and m's should be prohibited next to each other in captchas
like this!)

\- MCruncy (I'm really confident of my interpretation, but are capital letters
supposed to be reported as capitals on this captcha system, or is this one of
the systems that we're only supposed to submit lower case characters even when
the character that we're viewing is upper case?)

------
Keyframe
Now that programs are better and better at solving CAPTCHA - that means that
correct CAPTCHA input will mean the opposite from what it means now. Since
programs are better at solving CAPTCAH than humans, correct input (3/3 for
example) will mean it's a robot. Thus, CAPTCHA becomes relevant again.

~~~
return0
The endgame solution is to let humans judge other humans. In 5 years, you will
be interrogated every time you post to HN. The machines are here to make our
lives hell...

~~~
Geee
The question is: does it matter whether the participants are human if they are
intelligent enough to contribute. I think we'll eventually end up in situation
where captchas actually filter out humans :)

~~~
jordanthoms
Yeah, I've had that weird experience of spambots taking text excerpts and
posting it to avoid detection, but it's actually relevant information...

------
jrochkind1
From that caption "CAPTCHA images correctly solved by the algorithm", there
are at least two of them that I'm not sure _I_ can correctly solve on the
first try.

Which is generally my experience with captcha's these days, I only have about
a 50% success rate.

CAPTCHA is a failed strategy, time to give it up.

~~~
notatoad
it's a failed strategy for deterring spammers, but apparently reCAPTCHA has
done a good job of training google's computer vision algos. which probably
have better things to do than try to beat captchas.

------
zobzu
The program solves captcha that I, as a human, cannot solve. Pretty sure that
means captcha of that type are definitely dead.

~~~
Blahah
Same here. Over the last few years captchas have been getting more and more
difficult for me to decipher. It often takes me two attempts to get them
right. Particularly annoying on google groups when you need to add a lot of
members and you have to do a captcha for every ten members you add.

Whatever replaces captcha, I really hope it's less frustrating for the actual
humans

------
jere
>In this paper, we show that this system is able to accurately detect and read
difficult numbers in Street View with 90% accuracy.

> Turns out that this new algorithm can also be used to read CAPTCHA
> puzzles—we found that it can decipher the hardest distorted text puzzles
> from reCAPTCHA with over 99% accuracy.

Am I missing something or could we improve CAPTCHAs by mimicking street
numbers?

~~~
Kronopath
reCAPTCHA already does this, actually. It's part of the way Google trains
their computer vision algorithms to recognize numbers.

[http://techcrunch.com/2012/03/29/google-now-using-
recaptcha-...](http://techcrunch.com/2012/03/29/google-now-using-recaptcha-to-
decode-street-view-addresses/)

~~~
eurleif
But that's, as you said, to train their computer vision, not to make the
CAPTCHA better as a CAPTCHA.

------
ilitirit
To be honest, I can't even solve those reCAPTCHAs on that page (that's one of
my biggest gripes about reCAPTCHA). I think we're nearing a point in time
where if some(thing) can solve a particularly hard CAPTCHA, we can safely
assume that it's not human.

------
dlsym
"CAPTCHA images correctly solved by the algorithm" \- Ok. Now I have to
consider the possibility of being a machine.

------
mrsaint
Captchas were meant to keep spammers at bay. Unfortunately, that's no longer
the case. Thanks to "cloud technology" like DeathByCaptcha - that is, people
in countries where labor is cheap solving captchas all day - spammers have no
problem getting through reCaptcha-protected sites and forums to do their
mischief.

As a result, reCaptcha & co tend to be more of an annoyance to honest visitors
than to spammers.

~~~
t0lk
I have a mediawiki wiki and let me tell you there are tons of old spam bots
out in the wild, ones that would fail at the presence of any captcha. Come to
think of it, I've experienced most generations of spam bots since I started.
With no protection I'd get spam links/registrations every day. With a simple
question like 2+2=? I'd get one every 3-4 days or once a week and with
reCaptcha I'd get them a little less frequently. I had to switch away from
reCaptcha a few years back however when it appeared to no longer prevent spam
(people are paid to solve them all day, etc).

------
daffodil2
Wait, it's not clear to me from the blog post. Did they make a system that
obsoletes reCAPTCHA? If so, it's just a matter of time before the spam systems
catch up, correct? If so, what's the successor to CAPTCHA? Or is the web just
going to be full of spam in the future?

~~~
fuelfive
Digging in to the journal article, the technique they use can only scale to
captchas with 8 characters or less, so having a longer word is a simple fix.

------
varunrau
I've always felt that it would be only a matter of time before computer vision
would be able to solve the (re)CAPTCHA problem. Especially since digit
classifiers are able to match the performance of humans.

One approach that I enjoyed seeing was the use of reverse captchas. Here you
pose a problem that a computer can easily solve, but a human cannot. For
instance, if you ask a simple question (1+1=?), but you place the question box
off the screen so the user can't see it. A computer would be able to easily
answer the question, but a human user would have no way of doing so.

------
EGreg
Basically consider why we want to eliminate computers from accessing the sites
-- because we want to make account creation expensive, to prevent sybil
attacks and giving away scarce resources.

What is expensive? Reputation. That's where credit money's value comes from.

I wrote a more comprehensive piece here, in case anyone's interested:
[https://news.ycombinator.com/item?id=7601690](https://news.ycombinator.com/item?id=7601690)

------
shultays
My accuracy is way below 99%, good job Google!

Seriously though, I hope this does not mean there will be harder captchas,
current ones are already stupidly hard

------
msvan
Here's a captcha idea: make people write a 100-word essay on a specific topic.
If it's good, you're accepted and you won't have to do it again. If it's bad,
you're either a computer or cheap Nigerian labor. When we get to the point
where we can't distinguish a computer from a human, we'll just let them be a
part of the community.

~~~
epi8
Who judges if the essay is good?

------
aljungberg
Google software could use their 99% successful algorithm to filter potential
captchas. Then show the 1% they can't crack to humans.

Now the race becomes who can write the better captcha solver, Google or the
spammers? As spammers learn to identify things in the 1%, Google will
hopefully improve faster and continue to narrow the "hard to solve" band.

------
vfclists
Google are getting too creepy for any sensible persons liking. Addresses which
are off the street in apartment complexes are now getting recognized as well.

Whenever I see these kind of captchas I switch to audio captchas. It is rather
unethical for Google to use recaptchas in this way.

------
spullara
Reminds me of a hack day at Yahoo where one team made a captcha where you had
to match a photo with its tags and another team made an algorithm that would
assign tags to a photo. Both based on Flickr humans meant that the captcha was
easily solvable by the algorithm.

------
spullara
So, now if you get the captcha right you're a computer, otherwise you are a
human?

------
drawkbox
99% is better than most humans captcha accuracy. Back in my day humans could
still beat computers at Chess but nowadays computers can beat humans at
Jeopardy and drive. Interesting to see when it fully crosses over.

------
pavelrub
This is essentially the technology that was discussed here 3 months ago [1],
and it links to the exact same article on arxiv, titled: "Multi-digit Number
Recognition from Street View Imagery using Deep Convolutional Neural
Networks". [2]

They new addition to the article is that now they have tested the same type of
NN on reCAPTCHA, and (perhaps unsurprisingly) it works.

[1] -
[https://news.ycombinator.com/item?id=7015602](https://news.ycombinator.com/item?id=7015602)
[2] - [http://arxiv.org/abs/1312.6082v4](http://arxiv.org/abs/1312.6082v4).

------
pestaa
Folks, I figured it out! Let's use captcha so that visitors can prove they are
robots! If you fail these captchas, you must certainly be a human!

------
tsenkov
It's fascinating how, arguably simple software now, which is _the captcha_ ,
would inevitably become more and more complex as AI develops.

------
infinity0
Ironic how the HTTPS version force-redirects you to HTTP. (Amazon.co.uk
started doing this a few days before and it's pissing me off no end.)

------
plg
Why isn't google releasing the full algorithm?

~~~
dwiel
[http://arxiv.org/abs/1312.6082](http://arxiv.org/abs/1312.6082)

~~~
plg
nice! thx

------
aaronbrethorst
I'm impressed that their address identification algorithm can solve those
CAPTCHAs. I can't make heads or tails of them.

------
aviraldg
Isn't this expected (and a natural consequence of the fact that it's trained
on huge volumes of reCAPTCHA data?)

------
scorpion032
That is much more accuracy than with which humans can crack recaptcha.

I'd estimate my own accuracy rate to be 90% or less.

------
stuaxo
I'm sorry, as a human I have had to fill these street view style captchas in
all the time for google, so this is hardly a completely artificial
intelligence, humans have done it many many times, in fact I'm sure some of
the pictures in the articles have come up.

------
spcoll
It's a new success for Deep Learning. It seems to be actually 99.8% accuracy
according to their paper:
[http://arxiv.org/abs/1312.6082](http://arxiv.org/abs/1312.6082)

That's one order of magnitude higher.

------
blueskin_
Great... now they are going to get even harder to actually do.

------
northisup
Yet it says I'm a robot a good two of three times.

------
peterbotond
what if someone has bad eyes of some rare eye problem and can not solve
captcha problems at all? in other words fails captcha 90% of times.

------
zatkin
But can it beat CRAPCHA? [http://crapcha.com/](http://crapcha.com/)

~~~
Kequc
The crapcha widget actually looks a lot better than reCaptcha. I've always
lamented using reCaptcha just because of how ugly that thing is.

------
knodi
I just came here to say fuck reCAPTCHA! I hate it, I can't read it with my
human eyes.

~~~
ericcumbee
And you have added a lot to the conversation so thank you for that.

------
frik
Google's reCAPTCHA showed street numbers as one of the two catcha-"words" for
more than two years.

For me this was quiet annoying to input street numbers of others. It's a
privacy issue, it was like helping the NSA spying and one feels bad entering
Google's captcha.

What is even more astouning is that Google does not even mention all the croud
sourced "volunteers" that trained their OCR database. As Google use an open
OCR software (former HP OCR app from '95) it would be a good choice to publish
their data back to the community.

I removed Google captcha on my own sites and implemented my own traditional
captcha (on the first sight of it about two years ago).

------
rasz_pl
Does google aggregate&correlate data in vision algo?

For example for street numbers they not only have picture of a number, they
also have knowledge of all the other numbers on that street and guesses for
those other numbers. Easy to guesstimate order of a number by checking
neighbouring ones.

Same for book words, they have n-gram database.
[http://storage.googleapis.com/books/ngrams/books/datasetsv2....](http://storage.googleapis.com/books/ngrams/books/datasetsv2.html)

Thats a lot of useful MAP/ML data.

But the example they give for the new captchas all look like random crap,
"mhhfereeeem" and the like. Its like they are not interested in structure,
just pure geometry of letters/numbers.

------
rasz_pl
>CAPTCHA images correctly solved by the algorithm

well, isnt that great? Because I, HUMAN, can maybe solve _one_ of those (lower
right one).

I frickin HATE google Captchas and simply close the page if it wants me to
solve one, they are too hard for me.

------
exadeci
You're welcome google (we are the rats labs that teached their system how to
read)

------
leccine
And the cats, don't forget the cats!!
[http://www.wired.co.uk/news/archive/2012-06/26/google-
brain-...](http://www.wired.co.uk/news/archive/2012-06/26/google-brain-
recognises-cats)

------
conectorx
this is also can be done with tesseract or encog framework... i dont know
whats news about this

------
maccard
Damn, that's better than me!

------
sajithdilshan
Skynet

------
techaddict009
This is really Great. AI is getting really smarter and smarter day by day!

