
Eulerian Video Magnification for Revealing Subtle Changes in the World - clockwork_189
http://people.csail.mit.edu/mrub/vidmag/
======
fferen
I like how this kind of turns the "main idea" of computer vision on its head.
Namely, instead of trying to emulate and catch up to human abilities, it
augments them the way only a computer can; it plays to computers' strengths
instead of their weaknesses.

Also, who would have thought you can determine someone's pulse just by looking
at the color change of the face due to blood rushing in and out? If asked
beforehand, I wouldn't have even said it was theoretically (physically)
possible.

~~~
zher
Yes, it's hard to believe, but it's really possible to sense the small color
changes due to blood flow in the face using cameras. My earlier work described
how you just need a simple webcam
:)(<http://web.mit.edu/newsoffice/2010/pulse-camera-1004.html>). As a result,
I'm working on Cardiio (<http://cardiio.com>) to bring this technology to both
the web and mobile very soon. If you're interested in computer vision based
biosensing, please sign up!

~~~
aangjie
Just curious have you tried on people with darker skin colour/tan. I am trying
to imagine, what adjustments you'll need to make to detect on them?

~~~
AdamTReineke
There was a sample in the video, right after the white guy. The effect was
more subtle, but it was there.

------
jasonkester
There's a whole business to be made off that five second clip where they
amplify the motion of the sleeping baby's chest.

Anybody who's ever been a parent can remember going up to check on the baby at
night, standing over the crib for a full minute thinking "move! Move! MOVE!"
at the lifeless body of their baby, running all the horror stories they've
heard about cot death over and over in their head until finally the kid's
chest moves enough to breathe a sigh of relief and go back to bed.

The first company that adds this feature to a video baby monitor will make a
mint.

~~~
JoeAltmaier
A business preying off the fears of new parents, marvelous.

~~~
elemeno
There's a bit difference between building a business that preys on the fears
of a new parent and a business that acknowledges that new parents already have
fears and tries to reduce them. This I would say falls firmly into the later
category.

It's the difference between a tool which helps you make sure that your baby is
ok and one which tries to persuade you that if you don't use it you're
creating unconscionable risks to your baby which this product could save you
from. The former is a tool which is good, the later is one which preys on
parents.

~~~
JoeAltmaier
Totally disagree. The two devices are one and the same.

Checking on your baby in the middle of the night because you woke with a
sudden fear - normal parent anxiety, makes a funny story later.

Providing a device to allow constant electronic monitoring - validating an
irrational fear, reinforces anxiety, feeds into an obsessive loop, invents
issues for new parents just when they need to be calm and rational.

~~~
elemeno
We clearly disagree on a fairly basic level.

I'm very much of the opinion that anything which causes peace of mind is good
thing and teaches the parents that their fears aren't grounded in reality. I'd
say that most the fears that new parents have are rational in the overarching
sense (in that it's entirely rational for a parent to worry about about their
baby's health whenever the baby seems to deviating from the expected), but new
parents lack the knowledge to determine if any one particular event is
something that constitues an actual emergency, or is perfectly normal. Babies
commonly have minor health problems (such as colic or jaundice) when they're
very young which don't require medical attention, but sometimes they do
require attention and it can be hard to tell which is which since babies only
have very limited ways to express that they're suffering. A tool which
provides more information to the parents (like, say, being able to see that
the baby's heart rate or respiration has dramatically changed and so they
might not be crying as a result of colic, but because there's something else
wrong) and let's them make a more informed decision is clearly of benefit and
is by no means 'validating an irrational fear'.

~~~
JoeAltmaier
Peace of mind is most often effected by turning the mind from obsessive loops.
Selling expensive gadgets to nervous parents is way too tempting a profit
center to be responsibly marketed. How about by prescription only? That would
be a reasonable compromise. Ideally someone objective could be put into
picture, to prevent abuse.

~~~
jeremyarussell
I wasn't aware we had good indicators for SID.
<http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0002533/>

Lists some. Right after saying we don't know what causes it. The things listed
are pretty generic, like smoking around a mother expecting, etc. But nothing a
hundred percent. It's also the biggest cause of death in children under one
year of age.

Not sure where you are getting that it's something not to worry about. I do
wonder how many children you've had though.

Edit: Oh the point's made above were to beg the how would one get prescribed
something that so far seems pretty random.

~~~
JoeAltmaier
Oh, yeah, good point, attack me instead of the argument when you run out of
things to say.

I have 3 kids. I trusted in fate to take care of things I had no control over,
and was lucky they lived to be healthy confident adults.

I know of super-careful parents that raised precious children frightened of
the world. That loses something too - like the purpose of living.

I know engineers want to solve everything with an engineering solution. That
way leads madness. Next we'll be sticking monitors in the womb, because, you
know, sometimes the foetus doesn't develop right and maybe we can do something
about that too.

------
zemaj
Wow, reading someone's heart rate from a video.

Imagine that plugged into Google Glass. Overlay someone's heart rate next to
their face while you're talking to them!

~~~
jtchang
Excuse me while I book a trip to Vegas.

~~~
reginaldo
I was thinking about the implications of using this technique to analyze e.g.
political speeches and try to catch people lying on the act. Your application
(winning on card games) seems very interesting too.

~~~
sixQuarks
It seems like this technology can be used on historical videos. It'd be
interesting to see the heart rates from past speeches where we know they lied,
to see if there is a good correlation.

~~~
aiscott
That is an awesome idea.

I imagine the drug atenolol would become popular among politicians.

~~~
ch
That, or the radio address will once again become immensely popular.

~~~
tinco
Until realize that even though the filters in this post have been applied to
videos. It will be just as easy to apply the filters to audio, surely again
resulting in a revealed heartbeat :)

------
wxs
If people find this compelling, they may also like this earlier work from the
same group:

<http://people.csail.mit.edu/celiu/motionmag/motionmag.html>

This other algorithm amplifies only motion, and the algorithm is less
elegantly simple, however its results are very interesting and fun to watch as
well. Make sure to watch the video at the bottom of the page

[http://people.csail.mit.edu/celiu/motionmag/VIDEO_siggraph_2...](http://people.csail.mit.edu/celiu/motionmag/VIDEO_siggraph_2005.mp4)

~~~
pbhjpbhj
I watched that second video
[http://people.csail.mit.edu/celiu/motionmag/VIDEO_siggraph_2...](http://people.csail.mit.edu/celiu/motionmag/VIDEO_siggraph_2005.mp4)
\- that must be incredibly useful in engineering analyses.

------
btipling
Amazing for many reasons, but what I really find impressive is that messing
with video encoding requires like a PHD in everything because of how complex
things like H.264 are. At least that's my perspective after trying to grok the
H.264 spec.

There are all kinds of optimizations and algorithms for network latency and
video sizes, etc. I don't consider myself a layman but I certainly do not have
the knowledge and skill to do anything at all with video.

Do you know what a Inverse sub-macroblock partition scanning process is? I
don't, not even after reading the part about it in the spec.

~~~
taliesinb
This really doesn't have much to do with what makes computer vision difficult
(or rather, interesting). I doubt very many computer vision researchers are
conversent with the H.264 spec.

Typically toolkits like OpenCV take care of codecs for you (which make them
quite annoyingly difficult to compile on different platforms).

(I speak as someone who had done commercial computer vision algorithm
development.)

~~~
joeconway
Unfortunately when they do (My dissertation is on motion analysis using H.264
compression artifacts) things get pretty complicated very quickly

~~~
taliesinb
What precisely is your dissertation about? Inferring motion from compression
artifacts? Exploiting the fact that many codecs have already done motion
estimation so that you don't have to do image flow _again_?

~~~
joeconway
Yes, exactly that. Then filtering them because they don't actually show
optical flow, they just show a lot of noise

------
newbie12
This technology is badly needed today for skin cancer screenings. We could
save tens of thousands of lives a year with better detection of cancers and
pre-cancerous growths. It is crazy that most screenings in 2012 are still 3
minutes a year with a harried doctor.

------
siavosh
From my experience with CV, one of the biggest unsolved problems and rarely
publicized aspect of the field is how much variable tuning is often required
to get good results. I'm curious how sensitive these demos are to different
environmental conditions and the robustness of the algorithm.

------
samirahmed
i wonder if this could be used for lie detection

~~~
tokenadult
_i wonder if this could be used for lie detection_

I was tempted to downvote, but this is a legitimate question, and I guess you
haven't had opportunity to learn about how bogus "lie detection" in general
is. I happened to attend university where some of the earliest rigorous
researchers on lie detection were active,

[http://www.fas.org/blog/secrecy/2006/09/david_lykken_and_the...](http://www.fas.org/blog/secrecy/2006/09/david_lykken_and_the_polygraph.html)

<http://images.salon.com/april97/news/news970410.html>

<http://antipolygraph.org/articles/article-018.shtml>

[http://www.psychologicalscience.org/observer/getArticle.cfm?...](http://www.psychologicalscience.org/observer/getArticle.cfm?id=2119)

so people in my town have known for a long time that purported lie detection
tests don't work, and indeed our state has long banned the use of polygraphs
or other purported lie detection devices for any courtroom purpose. Law
enforcement investigators in our state use actual interviews of witnesses and
suspects, and by state law all police interrogations must be videotaped. That
helps immensely in reducing mistaken conclusions from improperly conducted
interrogations.

~~~
bane
I'm actually thinking if it might be a good tool to detect distress in a
subject. Not lying exactly, but might provide insight into a subject's
emotional state non-invasively during questioning or interrogation. People
might become distressed for many reasons that may not just be simple lying.

Perhaps even a Voight-Kampff-like test a la Blade Runner/Do Androids Dream of
Electric Sheep?

~~~
taliesinb
It never made sense to me that an _manufactured android_ would be easiest to
detect using a psychology test instead of say, _X-raying them_.

~~~
JoeAltmaier
Androids in that case were fully human, just manufactured. The principle
differences were improved physiology and durable mental processes, which could
have occurred naturally.

------
emmett
Is there an easier-to-follow explanation of the math that makes this work? The
video went too fast for me to quite understand it.

~~~
taliesinb
According to the paper, it is really quite a simple idea.

Imagine a filter that one might use in an audio context that boosts the
amplitude of frequencies falling in a certain range. But now imagine applying
that filter to a timeseries that actually comes from the intensity of a
_single pixel_ over time.

This has the effect of emphasizing motions that occur over timescales that
correspond to the selected band of frequencies.

The reason this is true actually comes back to the Fourier transform -- you
can imagine the motion of part of a complex image to be composed out of the
linear combination of many simpler 'sinusoidal motions', each of which is
being amplified by the per-pixel filter.

~~~
tomp
What about motion amplification? For example, the baby breathing in the bed...
Aren't multiple pixels involved then?

~~~
ClintTorres
Definitely.

There are two streams of information being analyzed. First is color, observed
as one position changing properties from one frame of video to the next.
Second is position, which is a change in location of things within the field
of view from one frame of video to the next.

Both color and position changes can happen at the same time, but it's easier
to see how this information is used if you start off by treating them
separately.

For color changes, imagine video of a stationary person's face where the
center pixel of every picture is the center of the face. The color at that
center pixel changes from frame to frame. At frame 1 the red value (of an RGB
triplet) is maybe 200 (out of 255), and by frame 5 the red value is 203. This
is a temporal red shift of just 3 levels over 5 frames.

For position changes, imagine video of a beach ball in a still pool of water,
where the center of the ball changes from frame to frame. At frame 1, the
position is (in x,y) (512,512), and at frame 5, the position is (512, 515).
This is a spatial shift of just 3 pixels over 5 frames.

In both of these cases, the observed change is probably too smal for humans to
notice in casual viewing, so an exaggeration of some kind is desired. To make
the change noticeable, we'll take the change of signal (3 values over 5
frames) and "amplify" it. These modified ("amplified") values will be shown on
top of the original source video, resulting in an exaggerated change. If a
color shifted by 3 red levels, we can show that by varying the color by 30 red
levels. If a position changed by 3 pixels, we can show it changing by 12
pixels. We can choose the level of amplification best suited for observing the
changes we've isolated.

Now the really cool aspect of these data streams is that their resulting
"amplifications" don't need to be applied to the same data stream that they
were observed in. It's possible to take a position change, and represent it as
a color change. It's possible to take a color change, and stretch it out over
a longer time (over more video frames). Any combination of properties you like
can be used to represent your observed set of changes. Some of the most
interesting representations will be a mixture of different data streams,
represented as another mixture of exaggerations to the original video.

~~~
tomp
Hm... I get the color one, and the position one in theory... However, if you
have a real-life video, how do you determine where the "center of the ball"
is? How do you even define what an object is, what are it's edges, and in
which direction did it move?

------
riemannzeta
These guys are going to make millions on baby monitors alone.

~~~
pbhjpbhj
Maybe, does it work well in a darkened room?

Baby monitors should probably be IR based; then this would appear to be a
great addition allowing a pulse to be determined from the video feed.

Current baby monitors use a pad underneath to detect movement including
breathing.

Personally, we used the "you can hear them screaming half-a-mile away" method
of baby monitoring.

------
carterschonwald
This is pretty darn amazing! I'm so going to try to implement this set of
algorithms later this week! :)

------
psb
If someone wants to do a kickstarter to try and use this technology to build a
glucose meter, I will be the first contributor

~~~
th0ma5
:( unfortunately might not be entirely applicable to this problem... unless
some measure of blood density and hence probably glucose mix... but at least
at _this_ layer of filtering and reasoning, it appears to be about minute
amounts of motion... is there something about motion that could denote
glucose? trembling maybe? maybe a score could be derived using something that
borrows from this technique, but you'd have to put more into it than it seems
this technique specifically shows.

~~~
psb
I was thinking more skin color or possibly changes in the eye (if I remember
right eyes get swollen when blood sugar is high) Honestly, I don't know if
there are enough outwardly visible changes for this to even be possible. But
there's a lot of money out there waiting on the person who can figure out a
way to use a cell phone do something like this.

~~~
nhergert
Good idea, but apparently much money has been spent on it already. Excellent
article on the physics of all things non-invasive (but especially glucose):
<http://www.mendosa.com/noninvasive_glucose.pdf>

~~~
psb
Wow, I didn't realize this was something of a white whale. Still sounds like a
fun nerd project though so I might spend some time on it

------
RockofStrength
This type of technology would be a nice novelty item to implement into a
futuristic zombie story. For example, there could be a large battle between
humans and zombies, and the humans are equipped with pulse-magnifying glasses,
allowing them to identify friend vs. foe.

------
vibragiel
Instant Heart Rate for Android uses something similar to check the pulse using
the phone's camera.

[https://play.google.com/store/apps/details?id=si.modula.andr...](https://play.google.com/store/apps/details?id=si.modula.android.instantheartrate&hl=en)

~~~
stefantalpalaru
Even more similar: <http://www.whatsmyheartrate.com/>

------
twp
Finally, you can "enchance" videos, just like they do in the movies:
<http://www.youtube.com/watch?v=Vxq9yj2pVWk>

~~~
eridius
That was fairly ridiculous, but where on earth did that clip of rotating the
video come from?

------
Hitchhiker
The team's initial work on topic:

<http://people.csail.mit.edu/celiu/motionmag/motionmag.html>

------
theschwa
The Is He/She flirting with me App is just around the corner. I wonder what
new privacy issues will crop up with this kind of technology.

------
JacobIrwin
If anybody could help clarify a concept (a specific filtering method), from
video:

"we filter the sequence temporally using a "???" ...and boost the temporal
signal to generate a video..." (at 2min. 45sec in video)

What type of filter is being described (missing from quotes; "???") ? Bonus
upvotes for providing a URL to definition (detailed description of the missing
term).

~~~
noisebeuter
He talks about a "Temporal Bandpass Filter". Not sure about the "temporal"
part, but in DSP a bandpass filter is a filter (think Audio-EQ) that passes
frequencies within a specific range and rejects the rest. See
<https://en.wikipedia.org/wiki/Bandpass_filter>

~~~
JacobIrwin
Thanks much noisebeuter!

------
RockofStrength
I have a tangential question: What's the state of technology that would allow
one to see through the eyes of another? This would be useful for piano
tutorials, etc. I think it could be done with a separate eye-tracking device
and a camera located between the eyes.

~~~
groby_b
Uh, what? If you play piano, you're looking at the score, not the keys.

~~~
RockofStrength
Not necessarily; I usually don't use a score when I play. It would also be
insightful to see the 1st person perspective of a jazz musician playing from a
lead sheet, to see how his eyes jump between page and piano.

------
mistercow
The simplicity and elegance of this algorithm is really breathtaking. I'm
mostly thinking out loud here, but I _think_ you could use a related process
to do inter-frame interpolation for smooth time-scaling of video.

~~~
0x09
You can! <http://ompldr.org/vYWh5eA/test2.mp4>

This was made by resampling in time with a DCT. Problems with this: 1.
practically, it needs the entire movie in memory. 2. gibbs phenomenon in time
is _really_ weird. These could both be addressed by using short overlapping
windows, and then the process becomes just like typical sinc/lanczos
interpolation in time.

~~~
mistercow
Well, yes, you can interpolate in the frequency domain like that, which works
pretty well for resampling 2D data as well. However, it does not transition
shapes from location to location; instead, it looks like one frame is fading
into another. What's fascinating about this multiscale algorithm is that it
appears to actually exaggerate the motions rather than merely increasing the
contrast of the changes.

------
MPSimmons
Excellent. Integrate this into the Tricorder projct:
<http://tricorderproject.org/>

------
joelthelion
Does anyone have a link to the paper?

~~~
apu
It's on the page: <http://people.csail.mit.edu/mrub/papers/vidmag.pdf>

------
lurker14
I just downloaded an app[1] that measures heartbeat rate by taking video of a
fingertip.

I was stunned by the brilliance of the application, how ubiquitous video
sensors can replace other senses.

There's something else to discuss here: We have a recently published paper
from MIT, including mathematical details of the process; and at the same time
a stack of closed-source commercial applications already in the market with
slick interfaces. Are they using the same technique? Is one app-maker trawling
preprints for app ideas? Are many app-makers copying one pioneering app-maker?

[1]
[https://play.google.com/store/apps/details?id=com.macropinch...](https://play.google.com/store/apps/details?id=com.macropinch.hydra.android&feature=related_apps)
but there are many similar apps.

------
lurker14
This is a similar idea to the temporal arterty thermometer, which uses an
infrared camera:
[https://en.wikipedia.org/wiki/Medical_thermometer#Temporal_a...](https://en.wikipedia.org/wiki/Medical_thermometer#Temporal_artery)

The mathematical model for the sensor is probably different, though.

------
anigbrowl
I'm not seeing what the big deal is here. Extraction of differential
information from video frames has been standard in video editing and
compositing software for years. Grab a demo copy of After effects, folks.

~~~
taliesinb
This is kind of the stock-negative comment that I don't like about HN. It
would be redeemed if it had a link, but instead its just a look-how-cool-I-am-
that-I'm-not-impressed-by-this _actually-pretty-awesome-technology-demo_ ,
dammit.

~~~
tptacek
The commenter you are replying to (and downvoting) has done (IIRC)
professional digital video work, and written many detailed and helpful
comments on HN about digital media. The comment you're replying to isn't
snarky. Assume good faith. Maybe he's just mistaken? Or maybe he's right, and
the thread just doesn't like that?

Here are some best-of's to peruse before concluding that 'anigbrowl is part of
the problem with HN:

* <http://news.ycombinator.com/item?id=1875412> (MIDI)

* <http://news.ycombinator.com/item?id=995096> (3D film)

* <http://news.ycombinator.com/item?id=4011770> (Google Moog doodle)

* <http://news.ycombinator.com/item?id=1263289> (DV on DSLRs)

A couple suggestions:

* If a downvoted comment with no replies upsets you so much that you feel a need to pile on with a pointed negative comment, go to the bottom of the page, type the commenter's name in, click 'comments', then sort by 'score', and read the top 10 or so first. You may find the urge to pile on dissipates quickly.

* While it certainly isn't going to cause drama here ('anigbrowl being one of the more levelheaded contributors to HN, way more so than me), I think you'll find that comments like yours tend to read as personalizing disputes, and lead to embarrassing spats you'd be better off not attaching yourself to.

~~~
taliesinb
Fair enough. Please accept my apology, anigbrowl. You are correct, tptacek, to
suggest I should assume good faith.

Looking back, I think what riled me was the word 'folks'. As in, "nothing to
see here, folks", although he didn't actually say that.

