
Visual Turing Test - stared
http://turing.deepart.io/
======
chei0aiV
I got 9/10\. Chose the first one incorrectly.

For me, the giveaway for computer-generated images was that they don't focus
on the "subject" and have too much detail on the background.

For example, look at the computer-generated boats - they blend into the
background. It's hard to tell where the boat ends and the rocks begin, or
where the sails end and the clouds begin. Or look at the grey house in front
of the dude's face or the pipe behind the glasses lady's head, which bring in
too much attention but add nothing to the painting.

~~~
kang
Yesterday I was reading this blogpost
([http://maryrosecook.com/blog/post/scarface-prince-of-the-
cit...](http://maryrosecook.com/blog/post/scarface-prince-of-the-city-and-
pieter-de-hooch)) so as soon as I clicked this link I thought no way can a
computer learn to do what I read yesterday on its own!

I had read headlines about computer-generated images but had never seen them,
yet I managed 9/10, anticipating that an artist's technique is not just
relation between color etc. but a philosophy.

Edit: Also, it is not a turing test in true sense, since it's one sided. It
would be turing test if I tell them what to draw & then it's tough to
distinguish between what an artist & computer paints.

------
vph
This is not a Turing test. Please correct me if I am wrong, but I bet that
none of this was drawn by a computer. Most likely, those that were supposedly
"drawn by computers" were pictures drawn by humans but applied various type of
filtering (e.g. those by the deep learning algorithms).

~~~
nbrempel
What about a photograph with filters applied to it? Does that count?

~~~
pqhwan
Think what would really count as a "visual turing test" is if the computer-
generated images were really "generated" by a computer, not rendered from a
photo. Still, it's an interesting exercise in how much the rendering can make
the photo feel "opinionated"—as if you can sense the painter behind it; that's
the criteria I used for choosing, but I got a 7/10.

~~~
versteegen
> what would really count as a "visual turing test"

I disagree - nobody would say drawings/paintings (by humans) based on photos
or observations by eye are less legitimate than scenes completely imagined.

~~~
lazaroclapp
How about the following: The examiner can provide any picture of their
choosing to two subjects A and B, where one is a human and the other a
machine. At the end of a reasonable period, both subjects respond to the
examiner simultaneously with one drawing each, derived from that same picture.
In creating each drawing, A and B are both trying to convince the examiner
that they are human. The examiner then should identify which of the two
subjects is a computer and which is human. To better reflect the traditional
Turing test, the examiner should be able to repeat this process with different
pictures (keeping the identities of A and B unknown, but fixed) before making
their determination.

Interestingly, this is a Turing test that, when applied purely to humans,
doesn't require said humans to share a common language.

~~~
versteegen
That's a nice protocol.

I have to quibble though that the Turing test is meant to be a test of
intelligence, and this sort of task seems pretty different, although it may
actually be visual-AI-hard (to invent a new term). So may deserve a different
name.

~~~
lazaroclapp
You can call it a test of human-level aesthetic sensibility if you'd think a
highly intelligent alien species would be unlikely to share our aesthetics.
Although, arguably, a highly intelligent alien would be able to understand our
aesthetics to the point where they pass the test regardless...

------
Houshalter
I got 6/10\. I guessed that most of the things that looked "too real" were
based on photos with filters applied to make it look like a painting, and that
the abstract stuff was generated by humans. E.g. I was very surprised a
computer produced this:
[http://turing.deepart.io/f/0.png](http://turing.deepart.io/f/0.png) or this
[http://turing.deepart.io/f/4.png](http://turing.deepart.io/f/4.png)

But the opposite heuristic doesn't work either, because not all the computer
paintings are abstract. It mimics a lot of different styles and I am extremely
impressed.

~~~
dingo_bat
>I guessed that most of the things that looked "too real" were based on photos
with filters applied to make it look like a painting, and that the abstract
stuff was generated by humans.

I too used the same method and 5/10\. Looking at the other comments, it seems
people have better idea what art is than I do.

------
BasDirks
10/10\. Most of these computer "generated" pieces are not good art. This is
painted by a human who knows what it is to be human in the place that is
painted: [http://turing.deepart.io/t/7.jpg](http://turing.deepart.io/t/7.jpg).
This is just eh, shit?
[http://turing.deepart.io/f/7.png](http://turing.deepart.io/f/7.png).

I am convinced intelligence programming will tackle these issues some day. But
apparently not yet.

~~~
mathattack
I was only 3/10 - so I think it takes an eye for art to beat it. In my mind
this is like a chess program getting to 1200. Not enough to beat an expert,
but it's on to something.

~~~
josu
I'm not an expert by any means and I got 10/10

~~~
mathattack
Perhaps it's what we were looking for?

I suspect that we can be trained to spot fakes too to make it harder. Perhaps
there will be some co-evolution?

------
DanielleMolloy
Note that all of the computer generated pieces were created by this impressive
deep net algorithm (paper prepublished in 8/2015):
[http://arxiv.org/abs/1508.06576](http://arxiv.org/abs/1508.06576)

Somehow I cannot believe how fast they got the idea to create profit from it:
[http://deepart.io/](http://deepart.io/)

~~~
stared
Yeah, things move fast there: Paper (August) -> Open source code (September)
-> Deepart (November).

As a side note, for me the pace was one of the reasons to move from science to
data science (shameless plug: [http://p.migdal.pl/2015/12/14/sci-to-data-
sci.html](http://p.migdal.pl/2015/12/14/sci-to-data-sci.html)).

~~~
nl
Were you part of the DeepSense Right Whale Kaggle team? That was some good
shit.

~~~
stared
I work at DeepSense, and with these guys, but I didn't participate in this
team. I was shocked as well; and now I try to learn from them as much as
possible. :)

------
rayalez
8/10, but I think this is extremely impressive.

If I wouldn't know that some of them were generated by ANN, I wouldn't
question that like 90-95% of them were painted by a human. And I consider
myself pretty alright at painting.

These 2 are especially awesome:

[http://turing.deepart.io/f/0.png](http://turing.deepart.io/f/0.png)

[http://turing.deepart.io/t/9.jpg](http://turing.deepart.io/t/9.jpg)

~~~
zyxley
That second one would make for an amazing canvas print.

~~~
kidzik
The second one is actually painted by Francoise Nielly [http://www.galeries-
bartoux.com/fr/artistes/francoise-nielly...](http://www.galeries-
bartoux.com/fr/artistes/francoise-nielly.html#.VrZsz17L-DA)

------
kidzik
Here are the results so far
[http://turing.deepart.io:3838/](http://turing.deepart.io:3838/)

~~~
_yosefk
As a 10/10er, I think the fact that people tend to score less goes to show
that:

1\. The photos are low-res enough to hide the most ridiculous artifacts
produced by neural nets

2\. The machine-generated images are trying to emulate the likes of van Gogh,
not the likes of Leonardo (again hiding the extent of NNs complete inability
to understand what they're doing)

3\. Most people simply neither paint nor appreciate visual art, so this is not
as powerful as the Turing test! (One point of the Turing test is that a human
easily passes it, because all humans speak; they don't all paint.)

~~~
rubidium
9/10\. The low res made it harder to tell which one was computer generated,
but it was still pretty clear. Now if they could do it full size, I'd start to
be interested.

~~~
kidzik
[https://deepart-io.s3.amazonaws.com/hi-res/leon3.jpg](https://deepart-
io.s3.amazonaws.com/hi-res/leon3.jpg) \- hi-res version of the algorithm is
ready. Soon we will make the second edition of the test, stay tuned.

~~~
dbyte
this is great work kidzik. looking forward to see more about it. hi-res are
quite good, that even if in the uncanny they can be quite a success for
deepart lovers.

~~~
kidzik
Thanks! We teamed up with the authors of the original algorithm and Leon
figured out quickly how to improve resolution.

------
stared
BTW: "An artist or an ape?" test by Mikhail Simkin
[http://reverent.org/an_artist_or_an_ape.html](http://reverent.org/an_artist_or_an_ape.html)

~~~
fogleman
I got 100% on this one. I looked at all of them before making my selections
and found what seemed to be a pattern amongst the ape ones.

~~~
arrrg
Yeah, that seems like the trick. The ape ones are really very similar to each
other.

------
tluyben2
Knowing upfront that one is computer generated does not really work nor
showing it to an audience that has seen deep neural net pics for months on the
frontpage of HN. Showing them on the street in a non tech neighborhood and ask
which is the better/nicer looking painting without saying that one is done by
a computer should get some better results.

Prepare for some people actually getting angry about it once you tell that the
paintings they liked most are computer generated (from pictures but still).

~~~
pault
> Prepare for some people actually getting angry about it once you te that the
> paintings they liked most are computer generated (from pictures but still).

My girlfriend is a fine arts major and isn't speaking to me right now.

------
tiagobraw
Note that it is much easier for a computer to create a human-like painting
than to behave like a human. The Turing test requires a two-way interaction
medium.

~~~
Omnus
Exactly right - the Turing test as it was originally conceived is very hard to
implement (some argue impossible). To use the popular phrase, this is "not
even wrong".

------
Anm
9/10

The computer does not duplicate selective detail. Artists will put detail in
areas of focus, and intentionally obscure or abstract other areas. Also, there
is often an interplay between medium and message. To that last point, I'm
guessing there is still a human "artist" who chose the reference scene and
pose and artist style, even if they didn't paint it directly. Towards that, I
see this as a creative tool to be included in some future Photoshop, or
similar.

Also... Share on Facebook? "App Not Setup: This app is still in development
mode, and you don't have access to it. Switch to a registered test user or ask
an app admin for permissions."

~~~
franciscop
8/10, also got "App not Setup": [http://imgh.us/not-set-
up.png](http://imgh.us/not-set-up.png)

------
firasd
Got 8/10\. Of the choices, I was instantly sure that this one wasn't computer-
generated from a photo because of the unnatural dark area on the top:
[http://turing.deepart.io/t/6.jpg](http://turing.deepart.io/t/6.jpg). I think
(even if photos weren't involved) the fact that the dark area is out-of-place
but still aesthetically congruent may suggest something about human creativity
that would be hard to replicate using a computer.

~~~
Scarblac
Actually I thought that was human because the nipple stood out so clearly.

------
ThomPete
It looks more like these images are procedurally made rather than actual AI
created paintings.

------
cooper12
Knowing that it was a turing test actually made it easier for me. (8/10) While
I did take into account the painting style (the random rainbow look was
suspect), I mostly ignored images where the subject looked like a photograph.
This one threw me off on that basis though:
[http://turing.deepart.io/t/4.jpg](http://turing.deepart.io/t/4.jpg).

~~~
chei0aiV
For me the giveaway was the motion effect of the hair following the tilt of
her head, followed by the spiralling background.

------
jeffreysmith
I recently played around in this space ( [https://medium.com/data-
engineering/artificial-startup-style...](https://medium.com/data-
engineering/artificial-startup-style-437f6090b1f7) ). I was merely thinking
through what it would take to turn this into a Turing test and not actually
executing one.

These guys seem to be trying harder to actually make this a Turing test, but
as the commenters above have noted, the problem with using source photographs
makes this a fairly imperfect test. I have a suggested fix for that.

In my last effort in that post, I actually used one source photograph, and
then compared the output by a human painter and the AI painter. My original
results were terrible, but then quickly vastly improved by a better
implementation of the algorithm, the Deep Forger implementation.

I think that this variant in the procedure, having both the human and the AI
start from the same source photograph, could make for some interesting
variants on Turing testing.

------
js8
I wasn't very successful, but I tried to use "uniformity" criteria - if it
looked like there are portions of the picture that were being omitted or
abstracted away, I assumed human and vice versa.

I wonder if the algorithm could be improved if it had a sense of what is
"important" in the picture, and then choose a different algorithm to process
important and unimportant portions.

------
_yosefk
I got 10/10 (no Facebook or Twitter account to share this, unfortunately.)
_Update:_ my wife got 10/10, too.

Right now neural nets do pretty poorly with "loose/messy" styles (you get
noise instead of brushstrokes; if they posted high-resolution images there, I
cannot believe anyone would score less than 10/10.) Neural nets do
_exceptionally_ poorly with "neat" styles (try copying the style of the Mona
Lisa into your photo and it'll paint an eye into your mouth.)

However, I'm pretty sure that an algorithm hand-crafted to replicate an
artist's style might fool me (just like a hand-crafted chess algorithm wipes
the floor with a neural net plus some tree search; not so for Go, I know - I
don't even play Go so I don't have a firm opinion on that one.)

I must say that this whole "photo + style" thing really gets my goat - not the
research as much as the sort of press coverage it gets. I project that a stock
market crash will improve such coverage significantly.

------
stordoff
I'd be very interested to see the source images before processing, to see how
the resulting images differ from the originals. FWIW,
[http://turing.deepart.io/t/8.jpg](http://turing.deepart.io/t/8.jpg) has a URL
visible in the middle.

One thing I did find interesting is that I did the test initially and got 4/10
(most felt like guesses to be honest). I repeated the test before looking at
the answers (though obviously knowing the score introduces some bias), but
zoomed the page so the images were about 2-3x bigger than the default. My
score increased to 9/10, and I was much more confident of my answers whilst
selecting them.

------
stephanheijl
9/10, but I was already familiar with the algorithm. Useful features to find
the nature of the "artist" for me included the following:

\- Is the style of the painting applied uniformly over the whole image? Humans
will, for effect, leave parts of a painting less emphasized and developed,
whereas this algorithm will generally apply a style to the entirety of an
image.

\- If a painting has a surreal style, the subject is generally also surreal.
Human painters distort shapes and forms. This is not done by this algorithm.

\- Humans will add contrast to make objects stand out, even if colours are
similar to their background. This is something the computers haven't yet
completely figured out.

Still, this is a very impressive algorithm.

~~~
dmalvarado
8/10, on a mobile device, just going on gut instinct

------
sawwit
I somehow doubt that this is just a photo stylized by a computer. Aren't the
eyes way out of proportion?
[http://turing.deepart.io/f/0.png](http://turing.deepart.io/f/0.png)

~~~
alphapapa
Yes, exactly. While the background in the picture of the baseball player
doesn't look photographic, at least the proportions are realistic. But the
proportions of the female face are completely unrealistic. How could it be
based on a photograph of a real human?

Are these computer-generated images based on photographs of real subjects, or
on photographs of stylized paintings?

I don't think this test is very interesting until we can see the images the
computer-generated ones were based on.

------
kailuowang
I got the 8/10\. Style is extremely important in understanding an artist and
his work. Without a grasp on the original style, it's hard to see if a
painting is genuinely of that style. So a more substantial visual Turing test
would be a series of original works by an artist v.s. a series of simulated
works targeting that style by AI.

------
obelisk_
9/10\. I could tell from some of the pixels and from having seen quite a lot
of computer generated imagery in my time.

------
jkxyz
10/10\. My partner got 5/10, which he claims is because he's "more like a
computer" (he's autistic). There were features in all of them that struck me
as expressive in a very human way.

It's amazing to live in a time when questions about human identity and
expression are more than just hypothetical.

------
mesozoic
9/10 I think you can somewhat tell by the visual flourishes that a real artist
will add to their painting that a computer will not. Also the computer ones
while not completely obvious I could tell were generated from a real picture
without exaggerations a real artist may add.

------
kaugesaar
Got 10/10\. I think though if I would have only be looking at one picture at
time and were to say yes or no. My score would have been much lower. Quite
often in the computer-generated ones it looks like someone have gone nuts with
the smudge tool in photoshop.

------
ipsum2
9/10, missed the last one. I feel like the main difference between the
generated one and the drawn one are:

\- obvious artifacts in the neural networks

\- humans paint light differently than how a camera captures them, evident
even after heavy processing.

------
supercheetah
Apparently I'm a computer since I only got 3/10\. Am I the only one that seems
to have a lack of artistic aesthetic?

I do try to appreciate art, but I'll readily admit that I just don't get a lot
of it.

------
PedroBatista
Your result is 10/10!

I guess as of today T-1000 would get his bitch ass kicked.

------
mikeash
I got 8/10 right. I didn't really think about how I might figure it out, but
just went with my gut feeling. Apparently that worked pretty well. I wonder
why!

------
renox
3/10\. I wouldn't call this a 'turing test' though, in a turing test I can ask
questions, here it's totally static.

------
ibic
Well, is it really considered a Turing Test? I thought in Turing Test, the
interrogator is the one to ask (choosing from questions at his will).

------
Vervious
5/10 – I'm not human?

------
danr4
does anybody know a way to run an algorithm with similar results on my own
pictures? or how kids are calling it: "is there an open source version of
this?"

~~~
stared
The easiest way is... [https://deepart.io/](https://deepart.io/). Open source:
[https://github.com/jcjohnson/neural-
style](https://github.com/jcjohnson/neural-style) (not sure if exactly this
one, there are a few implementations).

------
eva1984
9/10 first one incorrect

------
Ono-Sendai
Your result is 10/10!

------
WA
Well, this is really nonsense, unfortunately.

First, all images could've been painted by a human.

Second, if I always click the left one, it says _Your result is 5 /10_. A
turing test is to test "a machine's ability to exhibit intelligent behavior
equivalent to, or indistinguishable from, that of a human". If I – presumably
a human being – score 5 out of 10, the test is not a Turing test after all,
because humans aren't tested in a Turing test.

Edit: My observation was obviously wrong. Thanks for pointing it out.

~~~
mnx
You are not being tested, you are the judge. now, if all the judges do no
better than chance, the system has passed. I got 8/10, and knowing the deepart
pictures from before, I was pretty confident in some of the judgments, so I
don't think they pass.

~~~
kidzik
Indeed. We haven't pass yet - that's the first attempt though. Still only 66%
of human accuracy is quite remarkable.

