
DAIN: Depth-Aware Video Frame Interpolation - ArtWomb
https://grisk.itch.io/dain-app
======
Sesse__
There's a huge amount of research within this field. For a ranking list on a
standardized test set, you can see e.g.
[http://vision.middlebury.edu/flow/eval/results/results-i1.ph...](http://vision.middlebury.edu/flow/eval/results/results-i1.php)
(there are many others). DAIN is roughly in top 10.

Sadly, there's next to no focus on the realtime use case, which is useful for
e.g. sports. For my own project Futatabi (see
[https://nageru.sesse.net/](https://nageru.sesse.net/)), I chose to
reimplement one that's fairly obscure, DIS (Dense Inverse Search), on the GPU,
so that I could get it realtime. IIRC, it's nowhere as good as DAIN is, but
there's that 60 fps demand... :-)

~~~
dharma1
good info! How much better are the new deep learning based methods compared to
older optical flow approaches like Twixtor?

~~~
Sesse__
I've never actually tried Twixtor myself, but assuming Twixtor hasn't just
adopted the DL methods, I would assume they're significantly better for the
difficult cases. Especially occlusion and non-uniform flow (think small
objects that move the opposite way from their background) are really difficult
cases to get right with optical flow. You'll inevitably end up with some kind
of artifacts. The good news is that if you're only upsampling by 2x, half of
your frames are going to be original frames, which masks away most artifacts
unless they're really bad.

Do note that some of the DL stuff uses optical flow internally in some form.
It turns out that just hallucinating pixels directly isn't necessarily giving
the best possible results, but you can do stuff like putting an optical flow
layer in your network (it's differentiable), or doing optical flow first and
then using that as an input to a DL network. It's a good reminder that
designing your network for the actual problem gives the best results.

Generally I haven't kept track of the state of the art over the last year or
so; the field is just too big, and I just had to go with a method that was
implementable (especially without having tons of training data ready). I
tested 5–6 different ones that I thought could reasonably approach real time,
chose DIS, spent a while implementing it and then called it a day.

------
Animats
Oh, like Framefree, from 2006.[1][2] Framefree worked on harder cases, too;
not just video game characters with nice sharp edges.

That was a really good idea. It came from Kerner Optical, which was spun off
from Lucasfilm and went bust. They had their own frameless video compression
format, and there was a Firefox plug-in for playing it. Some of that
technology went into systems for making 3D movies from 2D movies, and ultra
slow motion for sports. I saw it up at Kerner, in Marin north of SF, about 15
years ago, and I had the plug-in and some files it could play.

It's basically an automatic delaminator (the hard part) that separates an
image into multiple planes. Then each plane from two images in a sequence has
morph points found. The interpolation is a standard 2D morph. You can morph as
slowly as you want.

The business end of the operation was totally botched - there were holding
companies, an offshore shell company in Jersey, something called Monolith, and
some company called Quin Land in Japan. All of which seem to have sunk without
a trace. Sad.

I wonder if anybody paid the maintenance fees on the patents.

[1]
[https://web.archive.org/web/20081216024454if_/http://www.fra...](https://web.archive.org/web/20081216024454if_/http://www.framefree.com/)
[2]
[https://www.youtube.com/watch?v=VBfss0AaNaU](https://www.youtube.com/watch?v=VBfss0AaNaU)

~~~
jedimastert
> Framefree worked on harder cases, too; not just video game characters with
> nice sharp edges.

Check out the claymation example:
[https://www.youtube.com/watch?v=eAUn7Nvx73s](https://www.youtube.com/watch?v=eAUn7Nvx73s)

~~~
foota
What's with the artifacts down the middle? Is one side 60 fps and the other
not?

~~~
afrodc_
Yes. It seems like the left is without the effect and the right shows the
improvement with the technique.

~~~
foota
Ok, thanks!

------
ArtWomb
Original research:

[https://sites.google.com/view/wenbobao/dain](https://sites.google.com/view/wenbobao/dain)

------
sabalaba
Very impressive work. I would love to see what this looks like when you start
to drop more and more frames. I’d be curious to see how this technique starts
to degrade as the input FPS goes down.

Seems like it already can dramatically reduce the amount of work involved in
making an animation. Very cool to see pixel art animated at 60 FPS!

