
Nvidia Can Artificially Create Slow Motion That Is Better Than a 300K FPS Camera - jedberg
https://motherboard.vice.com/en_us/article/ywejmy/nvidia-ai-slow-motion-better-than-a-300000-fps-camera
======
joshvm
Title is misleading. Nvidia have created a network which can slow down (i.e.
interpolate) videos by a factor of four.

Actual link: [https://news.developer.nvidia.com/transforming-standard-
vide...](https://news.developer.nvidia.com/transforming-standard-video-into-
slow-motion-with-ai/)

Most importantly, the footage shot in the video was at 2500 fps native [1] -
at 300k FPS you have to use a significantly reduced image size. FPS is
bottlenecked by the time it takes to read out the sensor, which is linear in
the number of rows in the image. So if you read out half the sensor, you can
double your frame rate. You can exploit this to get thousands of frames per
second on a cheap machine vision camera.

The camera framerate is largely irrelevant. While you can slow down footage by
four times (neat), that doesn't mean you can get 300k fps footage out of a
compact camera. The great benefit of this technique is really that you can
convincingly fake super-slow motion without having to sacrifice sensor
resolution. I can easily see this as a cool iOS or Android feature.

Maybe if they'd actually used footage that was at the limit of camera
technology this would be more impressive (I don't see a technical reason why
not?). Why not combine it with super resolution too? Interesting security
implications, as cameras with frame rates higher than (I believe) 1M fps are
ITAR controlled.

[1]
[https://www.youtube.com/watch?v=5mZovjRlkWs](https://www.youtube.com/watch?v=5mZovjRlkWs)

~~~
dvfjsdhgfv
> as cameras with frame rates higher than (I believe) 1M fps are ITAR
> controlled.

I'm curious what's the reason for that. I can understand that extremely high
resolution could be used for intelligence purposes/high quality satellite
images, but high frame rates?

~~~
joshvm
The UK government has a great website for this, where you can look up
restricted things and the reason for the restriction. Search for "frames".
Mostly it's because they're used for ballistics and weapons imaging.

They're listed under item 6A003 (broadly, cameras) on the export list which
includes:

\- Cinema cameras with frame rates more than 13,150fps

\- Mechanical/electronic cameras with frame rates more than 1M fps

\- Frame readout speeds of more than 125 fps full frame.

The last one is interesting, you can buy machine vision cameras with higher
frame rates fairly easily, so perhaps it's only something to worry about if
you're traveling.

[https://www.ecochecker.trade.gov.uk/spirefox5live/fox/spire/](https://www.ecochecker.trade.gov.uk/spirefox5live/fox/spire/)

------
contravariant
I've seen some exaggerated titles but this one is something else.

It's definitely good frame interpolation, but not only does the slow-mo
already look artificial at a mere 4 or 8 times slower than the original it
also does nothing to reduce motion blur, which makes it incapable of replacing
a high speed camera in any meaningful way.

I guess they may have meant that it can _also_ improve images that come out of
a 300k FPS camera, but that's a somewhat meaningless statement. If anything
more FPS makes things easier to interpolate.

~~~
ashleyn
Not to mention, frame interpolation isn't anything that inherently requires
complex AI. My old Samsung TV had this feature, and it created the irritating
"soap opera effect".

~~~
TomVDB
If your TV created the soap opera effect and this new method does not (does
it?), then the new method is clearly better and maybe it did require this
complex AI after all to make that happen.

~~~
mxfh
The soap opera effect comes from high FPS playback vs. the aqcuired taste of
the 24fps _cinematic_ framerate which is independent of the topic of slow
motion and is technically not a flaw, it's an aesthetic preference.

~~~
cjhopman
Actually, I don't think it's inherent to high FPS playback. It's also that the
capture is more fine-grained. I imagine you could capture 300fps, such that
each frame is captured over 30ms and play that back at 300fps and retain the
cinematic feeling that you are used to. I don't know if doing that would
actually bring any of the benefits that normal higher fps capture brings.

------
faitswulff
Some alarming spots - the ice skates on the hockey player, the planted foot on
the dancer - but otherwise it looks pretty convincing.

------
apazgo
Whats the big difference with this and [https://www.svp-
team.com/wiki/Main_Page](https://www.svp-team.com/wiki/Main_Page) ?

------
chimtim
This may appear to "work" but I doubt it is better than a 300K FPS camera by
any means. It will not work perfectly across all scenes, it cannot interpolate
uncommon movements and finally, it cannot interpolate many low-res cases like
5-20 FPS where there is actually significant missing information across
frames.

~~~
mxfh
That interpolates look to me significantly worse in case of highly volatile
volume affecting reactions, like the ballon and the the immediate jelly post-
impact sequence with the racket. There a clearly still hard limits to what you
can believably hallucinate without knowledge of the higher frequency processes
that have not representation in the source material and are to chaotic to be
learned from watching random 240fps videos for training.

Also for a research output the ground-truth real slomo vs. super slomo is
missing to make a proper quality assessment. The paper itself only list some
fairly low-frequency inbetween frames as reference for comparison with other
methods.

[https://arxiv.org/abs/1712.00080](https://arxiv.org/abs/1712.00080)

------
bassman9000
Clickbait. It needs the 300K camera input in order to create the output. edit:
misspelling

------
anonu
This is really cool. Seems like a better application of neural networks on
video than the deep fake stuff earlier this year.

------
hndamien
I can see issues with the blade on the ice skater. Very cool though.

