
Blade Runner re-encoded using neural networks - signa11
http://www.vox.com/2016/6/1/11787262/blade-runner-neural-network-encoding
======
teh_klev
This is the original article by Terrence Broad:

[https://medium.com/@Terrybroad/autoencoding-blade-
runner-889...](https://medium.com/@Terrybroad/autoencoding-blade-
runner-88941213abbe)

and its appearance on HN the other week:

[https://news.ycombinator.com/item?id=11766063](https://news.ycombinator.com/item?id=11766063)

~~~
dang
Thanks; forgot about that one.

------
svantana
This article alludes to, but never actually brings up, something much more
interesting than the project in question:

How can copyright coexist with human-level automatic analysis and synthesis of
works of art?

For example, Spotify is flooded with "covers" of hit songs that are made to
sound as similar to the original as possible. In my understanding, it doesn't
matter how similar it sounds, as long as it's "remade", it's considered a new
work of art. So what happens when the cover song process can be automated? It
seems likely that courts and lawmakers will need to define a sphere of
similarity around each work of art, within which all points are considered
copies.

~~~
buro9
Music has that sorted already.

There is copyright.

There is publishing right.

If you make a cover of a recording, it is the latter that you are involved
with. You'd have to go to the publishers of a piece of music to license the
right to publish your interpretation of the work.

Obviously this post simplifies, but it's already answered.

~~~
rangibaby
Re: publishing rights, you can make a cover version of a song using a
"compulsory mechanical license", whether the owner likes it or not.

~~~
ionwake
So are remixes allowed, or do you need permission from the original publisher?

~~~
squeaky-clean
For a remix, you would also need the permission to use the specific recording,
(which can be declined) as well as rights from the song owner.

For example, If you wanted to cover Viva la Vida by Coldplay, you do it and
pay the mechanical licensing fees. If you wanted to sample/remix it the
recording of Viva la Vida, you would need to contact Capitol Records, who
could decline or request specific payment. If you wanted to sample or remix
the performance of Viva la Vida from Super Bowl 50, you would probably need
the rights from CBS.

[https://loudr.fm/faq](https://loudr.fm/faq)

[http://www.ascap.com/playback/2011/01/features/limelight.asp...](http://www.ascap.com/playback/2011/01/features/limelight.aspx)

------
dogma1138
Can anyone explain what does this encoder actually does and how is it
different than any other encoder? The article almost seems to make it sound
that this encoder somehow reconstructed the film...

~~~
Mithaldu
This is possibly the next generation of video/audio/image codecs.

What he did was create a specialized compression algorithm that works very
well to compress the data that is each frame of Blade Runner, and decompress
it (lossily, like mp3) back into a video stream.

To put this into perspective: Blade Runner is 117 minutes long. At 25 frames
per second, that is 175_500 frames.

As he says, the input data he used was 256x144 with 3 colour channels, meaning
each frame was 110_592 bytes. This results in an input amount of 18_509 MB,
uncompressed.

His neural network compresses each image down to 200 floats though, i.e. 800
bytes. So the whole movie as compressed by the NN is 132 MB.

A friend of mine who works with neural networks estimated his NN to be roughly
90MB at 256x144, so the storage needed for the movie he created is about 222
MB.

This means that if this technology can be made fast enough, and the
reproduction high enough in fidelity, we could be looking at replacing
compressors (Fraunhofer MP3, Lame, MP4, DIVX, JPEG) made to handle _all_ types
of input equally well; with compressors that can are so good for one specific
set of data that compressor + data is smaller than anything the traditional
compressors could create.

And even if its not fast enough, it can still be a very efficient compressor,
trading size for CPU.

Plus, lastly, the NN itself can be compressed as well, and classic compressors
can possibly be applied to the frame data.

~~~
radarsat1
Except that in my understand, the decoded version looks like shit. So, while
optimistic that this technology might be quite good eventually, more
heuristics are probably needed for the right way to extract an optimal
encoding.

e.g. see the screenshots here:

[http://www.eteknix.com/blade-runner-gets-trippy-auto-
encoded...](http://www.eteknix.com/blade-runner-gets-trippy-auto-encoded-ai/)

~~~
lallysingh
As part of a compression suite, that may not matter all that much. I can think
of a hybridized version of this that starts with this ANN and then applies
'fixes' by a traditional video codec stream. If the ANN can represent most of
the bits of the frame pretty well, the ANN + Codec data can match a
traditional codec-only approach with substantially less data.

~~~
joe_the_user
The thing is that diff between the film and "encoded" version look pretty
fuzzy/muddy and so compressing that diff would have to be difficult.

Anyway, the article doesn't mention this idea, the first hn discussion had
someone claim some other ridiculous use of the fuzzy/muddy images that are
supposed to mean something and ultimately I'd say it's an example of "my
research project has cool images, I can use them for publicity".

------
nickpsecurity
The worst part about this encoder is it thinks it's dealing with static
pictures in plenty of cases. The version on the right simply has no movement
while movie continues to play on left. Also, faces don't move when one person
speaks. That's a no go even if the picture quality didn't look like crap.

~~~
Houshalter
It depends what your goal is. This is quite impressive for compressing each
image down to 200 bytes.

If you were actually using this for compression, this could be useful.You
could store the "diff" between the reconstruction and the actual pixels, which
would take many fewer bits than storing the whole image. Video compression
already does something like this, storing a frame, and then the following
frames just being diffs to the pixels of the previous frame.

The goal of this project seems to be more artistic, so the poor quality might
even be desirable.

~~~
nickpsecurity
It's not impressive to get an image small if you delete a bunch of critical
stuff out of it plus warp what remains. Ill agree on the art project part,
though.

~~~
Houshalter
What do you expect from compressing an image to 200 bytes? It's impressive
that it has as much detail as it does.

~~~
nickpsecurity
I expect that decompressing the image produces one that looks close enough to
the original to meaningfully call it compression. This I call mostly image
_deletion_.

------
klakier
Bullshit... this is still encoding. It does not matter that it is using neural
network to encode frame. Guys, when you get over this neural network hype? I
can as well create and encoder using some machine learning, to create a blurry
and inferior version of the movie. But what's the point? I could as well used
other prediction method with some kind of memory, much superior and nobody
will enjoy it. Summing up: nothing impressive here.

~~~
zodiac
> But what's the point?

Autoencoders don't really have much practical use now, but that's not the
point of them. The point is we really want to figure out how to do
unsupervised learning well and autoencoders are one of the few ways of doing
it.

We want to do unsupervised learning well because most learning that humans do
is unsupervised.

The idea behind autoencoding is that by forcing the network to try to learn
efficient ways to compress the data, it could learn important features of the
data. The fact that the pictures are so blurry means that this doesn't work
very well, but that's why it's a research problem.

Autoencoders don't work well, but some unsupervised techniques that extend on
them do, and we get impressive results like
[https://arxiv.org/pdf/1511.06434.pdf](https://arxiv.org/pdf/1511.06434.pdf)
(see page 5) where the network learns to generate natural-looking bedroom
images.

~~~
kastnerkyle
Note that DCGAN (or standard GAN generally) doesn't have an encode path (or
any other way to easily condition generation) so it is not well suited to this
_particular_ task. The image quality is stellar though, and I linked to
several recent papers above which combine a GAN approach with an explicit
encode. This should allow them to be used for the kinds of things in this
link.

------
ole_roy
But no audio, right?

So, you'd think that alone would be a strong indicator that there was no
intent (or capacity) to profit from the reconstructed version.

------
mhax
This reminds me very much of Parag Mital's work on audiovisual resynthesis:
[http://pkmital.com/home/](http://pkmital.com/home/)

IIRC he has been beset with takedown notices for his 'smash up' videos which
resynthesise the weeks #1 most viewed youtube video from the next 9 most
viewed videos [http://pkmital.com/home/youtube-smash-
up/](http://pkmital.com/home/youtube-smash-up/)

------
intoverflow2
The idea of this is way more interesting than the actual result. Unfortunately
it just ends up looking like something run through content aware fill rather
than anything interesting visually.

------
jegoodwin3
Discussions would be more focused if participants were aware of the AFC test
-- Abstraction-Filtration-Comparison

[https://en.wikipedia.org/wiki/Abstraction-Filtration-
Compari...](https://en.wikipedia.org/wiki/Abstraction-Filtration-
Comparison_test)

Will that test be applied here?

Does it need to be revised in light of deep learning?

------
annalewiss
These complexities and nuances of sci-fi culture and artificial learning were
quite possibly lost on whoever decided to file the takedown claim for Warner
Bros.

~~~
stefs
> whoever decided to file the takedown

maybe that was a bot, which would add another layer to the story.

------
timlyo
It's saying down for scheduled maintenance for me, anyone got a mirror?

~~~
kefka
Sure do.

If you're running IPFS locally use this link:
[http://127.0.0.1:8080/ipfs/QmPo9savMewcGCzUhzNQi96b6gSSoQ4k4...](http://127.0.0.1:8080/ipfs/QmPo9savMewcGCzUhzNQi96b6gSSoQ4k4DFEVkF51dCU83)

If not, use this one (public IPFS gateway)
[http://ipfs.io/ipfs/QmPo9savMewcGCzUhzNQi96b6gSSoQ4k4DFEVkF5...](http://ipfs.io/ipfs/QmPo9savMewcGCzUhzNQi96b6gSSoQ4k4DFEVkF51dCU83)

Be aware, there is no sound with this video. You'll have to add it from your
own sources (nor did I want to add the sound from my dvd set).

And frankly, IPFS is awesome :)

