
Interactive Dynamic Video - ne0phyte
http://www.interactivedynamicvideo.com/
======
spyder
Pokemon GO and Interactive Dynamic Video:

[https://www.youtube.com/watch?v=9f1fCCb3hVg](https://www.youtube.com/watch?v=9f1fCCb3hVg)

~~~
erdevs
Thanks for this link. Really cool use case, and maybe Niantic will check it
out, if they haven't already.

~~~
RIMR
Yes, my battery doesn't drain nearly fast enough when I'm playing. This would
definitely fix that.

~~~
erdevs
Funny comment. :)

But actually this is unlikely to make a big difference on battery drain.
Pokemon GO appears to be built on Unity and Unity is pretty notorious for
having poor battery management. Whether you're pegging CPU or not, Unity burns
lots of power. So, the incremental battery life hit is likely to be low in
doing image processing on top of a frame.

~~~
vanderZwan
Seriously, fuck Unity. It's the indy game equivalent of vendor-lock in, with
everything negative about it.

They can't be arsed to fix a simple VSync bug on Linux, meaning even simple
sprite games, or even more insulting: TIS-100, a game about _optimising
assembly_ on fake retro-hardware, deliberately made to look like a _terminal_
, always burns 25% CPU power on my laptop.

More importantly, as Paolo Pedercini[0] mentioned on Twitter recently:

> _I can open my Flash crap from 15 years ago no problem, but a project I made
> last year in Unity 5.1 crashes silently if rebuilt in Unity 5.3 I 'm telling
> you: today's indie game production is a Unity monoculture that will
> disappear at the next hardware architecture cycle_

[0]
[https://twitter.com/molleindustria/status/760904169444274176](https://twitter.com/molleindustria/status/760904169444274176)

------
dspoka
When I first saw this (extracting audio from visual vibrations):
[http://news.mit.edu/2014/algorithm-recovers-speech-from-
vibr...](http://news.mit.edu/2014/algorithm-recovers-speech-from-
vibrations-0804) it seemed incredible that the signal the cameras could
capture was strong enough. This work is even cooler, even more non obvious and
seems closer to wizardry.

------
gavinpc
What exactly are the implications of this for construction engineering, which
must already have some way of measuring vibrations?

I.e. does this measure something previously unmeasurable, or is it just that
the visualization provides an extra channel for interpreting the data?

Or is it that it would make this kind of tool available to anyone with a
digital camera?

~~~
tlb
The vibrational modes of a structure are a sort of signature. When it changes,
that means something in the structure has changed. Perhaps a beam has rusted
or buckled, or bolts have come loose, or concrete has cracked. This is
traditionally tested by placing accelerometers in the structure, but it might
be much cheaper if you could do it with a camera.

------
ne0phyte
Here's TED a talk about the technology
[http://www.ted.com/talks/abe_davis_new_video_technology_that...](http://www.ted.com/talks/abe_davis_new_video_technology_that_reveals_an_object_s_hidden_properties)

------
arxpoetica
The implications of this technology are staggering. He doesn't even talk about
online use-cases, but imagine (just as an example) if YouTube added this
algorithm to all it's video library, and started allowing you to interact with
videos in very different ways. Crazy stuff.

~~~
iKlsR
Yes, so many applications, stop motion would benefit from this incredibly.

------
jefe_
The consumer use cases are interesting but the propagandist use cases are
terrifying. Along with this (Real-time Face Capture and Reenactment):
[https://www.youtube.com/watch?v=ohmajJTcpNk](https://www.youtube.com/watch?v=ohmajJTcpNk)

~~~
olewhalehunter
Which country (or organization) do you think will be (or has already been) the
first to implement a program of creating physical mimics of humans for
military and intelligence purposes? With technologies like this and CRISPR,
video, testimony, or genetic evidence could be thrown out the window in
criminal or military cases.

------
jasonkostempski
Kids would go bonkers if they could record their toys and dance with them like
at the end of that video.

------
ilovecomputers
Signal Processing was that subject I found difficult and I found no reason to
be interested in audio filtering, so I didn't find an incentive to work
through that difficulty. Then I see applications of signal processing, like
this, that goes beyond simple audio filtering, that makes me want to learn it
again. Honestly, this looks like wizardry! It is no surprise, MIT is on the
bleeding edge of signal processing and it mainly is due to Dr. Oppenheim . He
wrote the textbook on DSP and runs the DSP Group at MIT.

------
Phithagoras
Papers of their findings can be found at
[http://www.interactivedynamicvideo.com/publications.html](http://www.interactivedynamicvideo.com/publications.html)

------
jcoffland
I wonder of you could use this on car commercials and thereby deduce the
actual comparative quality of construction. Assuming that better built cars
would vibrate less at speed. It would be interesting to understand where the
vibrations occur.

A question for the authors. All the video is with vibrating but otherwise
stationary objects. Must the object be stationary for these tools to work?

~~~
initram
Probably not because in a lot of car videos there's no real car. Here's one
simple example:
[https://www.youtube.com/watch?v=b7vTM4_rjhs](https://www.youtube.com/watch?v=b7vTM4_rjhs)

There's a company that makes a rig that's basically just an engine and 4
wheels. You film your entire commercial using it and they replace it with
computer imagery of your car in post. I can't find the video at the moment,
though.

------
a2tech
Thats the next best thing to magic that I've ever seen

------
spoiler
This does look fascinating. The video is a bit showbusiness-like (for a lack
of better expression).

A few questions if the author is on HN: I wonder how long it takes to analyse
a 5 second video? Also, it seems the algorithm only works with static images
after the initial video analysis, or am I wrong? Also, how long does it take
to render the "virtual" state of the object?

~~~
abedavis
Hey, first author here. Good questions.

The simulation runs in realtime, and is interactive. Generating the simulation
can take a while though.

The 5 second video took about a minute to process on my laptop. But longer
videos, and higher-resolution videos, can take much longer. For instance,
we've used the technique on high speed video to recover audible vibration
modes, and this can take hours to process because the video itself is so big.

I wrote the processing code in Matlab, and the simulation in c++ and GLSL. The
simulation is pretty well engineered because I wanted it to run in realtime. I
didn't put a lot of time or effort into optimizing the video analysis part
though, so it probably could be made faster.

~~~
spoiler
Hi!

Thanks for responding to my questions! The whole project is amazing and I hope
you keep working on this (and future projects) with the same enthusiasm as you
do now! :D

------
pimlottc
Wow, more incredible image processing stuff from MIT. What's really cool is
how it shows how much hidden information is out there there in world to be
captured within a normal camera. We just need to be aware of it and analyze it
properly. For me, it just re-injects a bit of wonder into the everyday world
we live in.

------
ARothfusz
This might make the VRML included in MPEG4 useful. Could possibly use it to
pass along the physical constraints and other data needed to make it
interactive. ref:
[https://en.wikipedia.org/wiki/MPEG-4_Part_11](https://en.wikipedia.org/wiki/MPEG-4_Part_11)

------
two2two
This would be fantastic for getting a feel for the tactility of a product
you've not yet seen in person.

------
johnhenry
I would love to see what this could do when combined with light field
technology
[https://www.youtube.com/watch?v=xNJZHFZEkYQ](https://www.youtube.com/watch?v=xNJZHFZEkYQ)

------
iamcreasy
So if we can use this algorithm for Structural Health Monitoring (like a
bridge shown in the video) - what's the major obstacle to replace all
destructive testing in mechanical engineering with this kind of algorithm?

------
radarsat1
Dedicated host name for a single algorithm, TED talk, high-production value
video with professional animations, narrator talking excitedly, more kitschy
examples of applications than description of the algorithm itself.

Yup, MIT.

~~~
abedavis
I am a grad student. If you consider how many hours I work, I probably make
less than minimum wage.

I am also the first author of the paper described in these videos, which my
colleagues and I published in the top academic journal in our field.

I also made all of the videos, did the voiceovers, and hand drew the
animations (though I'm flattered you think they were professionally done). I
wrote and presented the TED talk. I purchased the dedicated host name for a
single algorithm (using money from my own grad student stipend no less) and
created the webpage myself.

I am an academic first, and I take academic integrity very seriously. I also
take education very seriously. I consider educating the public about research
to be part of my job, and this is done best when people are excited about the
research.

Also, this work IS exciting. If it weren't we wouldn't have spent so much time
working on it and it wouldn't have been published in ACM TOG. But back to
flashy videos and press...

Consider that in the past three days, nearly 100,000 people (and counting) sat
through a video where I explain what vibration modes are. VIBRATION MODES.
They may have been lured in by pokemon, but kids who remember that video won't
have to ask their physics teachers "but why should I care about this? what is
it useful for?" Hell, if it gets people excited I hope teachers show it to
their students before they teach the topic. I'm not making money off of these
videos, I'm just stoked this many people are getting excited about research.
Our paper has 17 numbered equations in it - it's not exactly a page turner.

When scientists don't make an effort to communicate their work to the public,
that responsibility falls on people outside of the academic community - people
who often don't understand the work. When we make things harder for the press,
we only encourage them to bastardize the work to make it more palatable to the
general public. By taking an active role in how we present academic work to
general audiences, we can better shape the message, manage expectations, and
help prevent content from being sacrificed for click-bait.

Ok, I'm going to get off my soap box now. Cheers! -Abe Davis

~~~
erdevs
I think your work (and your promotion of that work!) is awesome! Please take
heart in all the appreciation and encouragement in this thread (and
elsewhere)!

~~~
abedavis
Thanks so much! I really appreciate that :) And, in fact, I'm not at all upset
about the original comment - I totally get where the skepticism comes from.
Polish and substance are too often uncorrelated when it comes to science in
the media. I think it's important not to assume they are anti-correlated
though, because then we start discouraging scientists from trying to reach
people beyond the academic community.

