
Learning to Cartoonize Using White-Box Cartoon Representations - gitgud
https://github.com/SystemErrorWang/White-box-Cartoonization
======
fxtentacle
While the results certainly are pretty, I don't see how learning has taken
place.

This cartoon filter still has the same issues as previous attempts, which is:

\- omitting borders that are semantically important but have a low color
gradient

\- not collapsing small areas into lines

For the first issue, there's an example image of a picnic on white background.
A human cartoonist would most likely draw a full outline around the white
spoon, because it is important for conveying the type of object that this is
supposed to be. With this algorithm, the spoon gets partially merged with the
white background and without the reference photo I would have a hard time
identifying it as a spoon.

For the second issue, kook at the photo of the Asian girl with patterned
skirt. A human cartoonist would most likely observe the regular grid pattern
and replace it with thin lines, thereby communicating that all of it is one
same thing. This algorithm, on the other hand, treats each tile of the pattern
individually, thereby making it look more like a crystal or crumbled foil.

I personally also prefer white-box algorithms, but there's no denying that
creating a cartoon requires a lot of prior knowledge about which features to
retain as important and which features to abstract away. As such, I see the
real challenge in somehow producing good saliency training data for millions
of images. I mean ideally you would want the 5 year video stream plus eye
tracking data of a baby starting to grow up...

~~~
thaumasiotes
> For the second issue, kook at the photo of the Asian girl with patterned
> skirt. A human cartoonist would most likely observe the regular grid pattern
> and replace it with thin lines, thereby communicating that all of it is one
> same thing. This algorithm, on the other hand, treats each tile of the
> pattern individually, thereby making it look more like a crystal or crumbled
> foil.

Something similar is going on with the photo of the merlion statue. The entire
body is scaled, and a cartoonist would definitely represent that. But (because
of lighting?), the algorithm renders the tail smooth instead of scaled.

------
xingyzt
This could replace all of the WikiHow illustrators that trace from stock
photos, and muddle the waters even more for their copyright status.

[https://onezero.medium.com/we-finally-figured-out-who-
makes-...](https://onezero.medium.com/we-finally-figured-out-who-makes-
wikihows-bizarre-art-6c5d69b71347)

~~~
enriquto
It's probably much harder to arrange people and objects to reproduce such a
scene than to make a basic wikihow-style drawing.

------
taneq
If I understand this right, they've built a 'cartoonify' filter to convert
real-world images into cartoon format, and then trained a neural net based on
these image pairs? If so, what does the neural net add?

~~~
ramblerman
Kind of agree, I think if they have a good way to create the underlying
dataset, a NN to go in the opposite direction might be more interesting.

~~~
amelius
This would be very similar to the "super-resolution" approach.

E.g.,

[https://medium.com/beyondminds/an-introduction-to-super-
reso...](https://medium.com/beyondminds/an-introduction-to-super-resolution-
using-deep-learning-f60aff9a499d)

------
xwdv
I could have sworn this kind of thing already exist in some apps. Prism?

~~~
JKCalhoun
Yeah, looks exactly like some of the Prism filtering I've toyed with.

------
dzink
The big question: Does copyright law apply to cartoon version of copyrighted
images? Transformative work can circumvent copyright law, but are you allowed
to feed copyrighted images into an AI algorithm to create cartooned versions?
Who owns the copyright at that point?

~~~
jrockway
I don't see any reason why copyright would not apply. If you take someone
else's photograph, change the white balance, and start selling copies of it,
that's classic copyright infringement.

As for input data to models, my intuition is that they would be tainted by the
copyright of the input images. It's just that nobody has a bot for scanning AI
models for their photographs, so you don't see a lot of litigation or DMCA
takedown requests here. It's easy when someone just uploads your photo to
their website. It's hard when the photo contributes some weights to a neural
network.

My main takeaway is that copyright is very imperfect. It doesn't allow for any
unsolicited enhancements of someone else's work.

~~~
ximeng
You'll get further issues once Getty Images automatically generates
copyrighted images covering as much of the latent space of interesting images
as possible.

~~~
rcxdude
In principle by the way copyright works this isn't really an issue: if an
image is generated independently then copyright does not apply. However, in
the event that you wind up with something similar, it becomes something which
you would need to prove (and people have lost cases because they could not).
On the other hand, such large-scale generation of images will likely be
treated differently by the courts than other means of production.

------
ashleyn
Be great to see this in a fragment shader someday.

------
thih9
I really like the cartoonized photos of detailed landscapes.

I think human faces and bokeh could use some improvement.

The former seems especially tricky; showing or hiding certain lines might
change the perceived emotion and result in a different image.

------
patel011393
To a certain extent, PowerPoint's "Artistic Features" option under "Picture
Format" allows for similar effects. The paintbrush options is like this
cartoon style. If I had the time, I'd definitely choose this cartoon program,
as PowerPoint's effect is not as clean.

------
amelius
I didn't even notice these pictures were "cartoonized" until I zoomed in on my
smartphone ...

If you make an imaging effect, please make sure it works at all scales. And
I'm also looking at you bokeh-folks ;)

------
ggm
Rotoscope?

~~~
gonzo41
yep, this seems exactly the same. It's also fitting the Keanu Reeves is in one
of the test images. Scanner darkly was a great film, time for a rewatch

------
wnkrshm
Those images that don't already work because of their colors and composition
have the charme of traced photos.

