
Few-Shot Video-to-Video Synthesis - ArtWomb
https://nvlabs.github.io/few-shot-vid2vid/
======
greggman2
If someone wants a business idea take this tech and use it to offer a service
to fix up old videos. At least for a few years (10-20) there should some
market. I recently paid of have some family vhs tapes converted to .avi files.
Those vhs tapes are originally conversions of 8mm film. This is what most of
the video looks like

[https://i.stack.imgur.com/tqPDO.gif](https://i.stack.imgur.com/tqPDO.gif)

Actually that's probably better than average. Enough to bring back memories
but pretty awful and hard to look at more than a few moments especially from
the flicker.

Seems like a perfect job for computational video processing

~~~
alexcnwy
The flicker would be a challenge but you can use something like DeOldify [1]
or super resolution [2] on the frame sequence to improve the colour and
resolution... although you may get some weird artifacts between frames using
image models rather than models built for video.

Generally video models are quite far behind image models because they're so
much higher dimensional (much more computationally expensive and expensive to
annotate) but I'm sure it's a matter of time before someone releases a
DeOldify type model for video (if one doesn't exist already).

[1] [https://github.com/jantic/DeOldify](https://github.com/jantic/DeOldify)
[2] [https://github.com/idealo/image-super-
resolution](https://github.com/idealo/image-super-resolution)

~~~
yboris
For super resolution of video, the best thing I've come across is _TecoGAN_
which uses adjacent frames for extra information, creating a consistent flow
between frames.

[https://github.com/thunil/TecoGAN](https://github.com/thunil/TecoGAN)

------
taneq
Deepfakes were just the very thinnest end of the wedge. We're only a few years
away from video not being a reliable proof of anything at all.

~~~
kkotak
I wonder how this affects the gaming industry.

~~~
dmos62
I hope it helps bring down world-building costs. Currently a lot of projects
are impossible to indie developpers simply because of the amount of manual
labour involved in modeling and texturing.

~~~
lasagnaphil
One of the effects of deepfakes is that it makes the inherent value of
simulating reality to the utmost detail vanish over time. Our whole mass
entertainment industry is built by worshipping the quality of simulation (for
example, those flashy VFX effects, realistic landscapes, and Lapunzel's long
hair.) Right now, those details are achieved by hundreds of animators, motion
capture actors, and technicians, and by having huge render/simulation farms.
When virtual realism becomes easily replicable (by regurgitating previous data
through a deep learning pipeline), the old values of simulating reality will
soon vanish over time, and therefore the indie artists would be able to
compete in a fairer level than the entertainment monoliths.

However, one of the problems of deep learning is that you need to have a good
dataset first, but how are you going to build one? Well, the big game
companies wouldn't reveal their models and textures that easily. For indie
artists to utilize this technology, there needs to be a centralized community
project built around gathering and preprocessing data, rather than waiting for
someone like Adobe or Autodesk do it.

~~~
dmos62
Good points. Same holds for any other industry where data is important. Data
is power, because it's legal to thwart competition by not sharing data. It's
plain to see how without regulation this leads to uncooperative data
behemoths.

------
hirundo
"The code is ready for release, but we're still waiting for lawyers to resolve
some legal issues. Once it's approved we can release it." \-- issue #1

Kudos to NVIDIA Corp. for the planned release of this code. It seems to have a
lot of commercial potential. Maybe they see it increasing demand for their
hardware.

~~~
fatjokes
Actually I suspect it's more because of the push in the research community for
improved reproducibility.

~~~
BubRoss
What are you basing that on? Having source tied to a paper is very common in
computer graphics and has been for a long time.

~~~
godelski
Really? I'm doing a Ph.D. in sci-vis and my experience has been that people
don't release code. One time I emailed a group asking if they were going to
open source their code (and asking questions about some parameters that were
unexplained in their paper) and a week later they responded by telling me to
go the tutorial page on pytorch (seriously.... just ghost me if you're going
to pull that kind of crap). It seems to me that a lot of groups keep code as a
"secret sauce" per-say. Personally I feel that's anti-scientific,
reproducibility is a fundamental element to doing science.

~~~
bonoboTP
Big and famous groups/companies usually do release their code nowadays.

I'm automatically suspicious when someone doesn't, even though I guess most of
the time it's nothing nefarious. It's just extra work and effort to bring the
code into presentable shape and that effort could be spent on the next paper.
Once the paper is published, the material benefits have been reaped,
publication count incremented. Of course this is a short-sighted view, because
in the long term not only the paper counts matter, but also one's general
reputation within the community.

------
Invictus0
I'd like to see these videos in higher resolution. I have to squint to resolve
any detail here.

~~~
p1necone
I imagine this would be much less convincing in higher resolution.

~~~
diesal11
The Paper includes some higher resolution examples which are quite convincing

~~~
p1necone
The paper is a PDF file, is there another video component I'm missing?

------
Mathnerd314
It seems to do motion transfer alright but the legs in the dancing video get
warped instead of moving so there's some frames where they have triangle legs
vanishing at the knee.
[https://youtu.be/8AZBuyEuDqc?t=34](https://youtu.be/8AZBuyEuDqc?t=34)

Maybe if you scale up the training data the model is going to learn some
better warping, but I think to really get better photorealism there has to be
a 3D component, some kind of distance/movement estimation like in
[https://arxiv.org/pdf/1704.07804.pdf](https://arxiv.org/pdf/1704.07804.pdf)
and a shape inference/transfer step.

------
aledalgrande
Why are researchers pushing deepfake?

~~~
TeMPOraL
They're not pushing deepfakes, they're pushing video-to-video synthesis.
Deepfakes is one of the many things you can do with it, and not nearly the
worst or most interesting (despite the media attention it got).

------
EGreg
In five years they will have adversarial networks produce hyperrealistic
videos of anything.

Can’t wait to see FOX News anchors deepfaked to have the exact opposite views.

