
First-Person Hyper-lapse Videos - davidst
http://research.microsoft.com/en-us/um/redmond/projects/hyperlapse/
======
UnoriginalGuy
The result is quite simply breathtaking. It looks like something shot for a
movie using a stabilised dollycam, the fact they were able to achieve the same
thing using nothing but a GoPro, their software, and likely a week of post-
processing on a high end desktop PC is simply amazing.

I hope we see this technology actually become readily available. There might
still be work to be done, but in general if they can reproduce the demo videos
with other content then they're on to something people would want.

~~~
pron
It's striking but it's far more believable when you realize that they need to
play at a much faster speed than the source, so they have tons of extra
samples from which to extract information. They basically use all that data in
the extra frames (that would otherwise simply get tossed away in a regular
time-lapse) to construct a 3D scene. This wouldn't look nearly as good if they
had to play it at normal speed.

~~~
bane
On the flip side, it sounds like shooting at higher fps and using this to end
up with a 24 or 60 fps result might end up with some cool results.

~~~
Osmium
Camera movement between each frame would be minimal though so there'd be a lot
of overlap between frames so minimal extra information. I'd guess the key to
improving this result would be multiple cameras at different angles; I imagine
it only works as well as it does because the GoPro uses a fisheye lens.

------
Mithaldu
Since it's not quite obvious, the supplementary page has videos with better
bitrate than what youtube did to them: [http://research.microsoft.com/en-
us/um/redmond/projects/hype...](http://research.microsoft.com/en-
us/um/redmond/projects/hyperlapse/supplementary/html/climbing_our_result.html)

------
rkuykendall-com
Interesting that the final video ( mostly the rock climbing ) resembles a
video game, where shapes and textures "pop-in" as they are rendered. The
technical explanation video was really well done.

------
msane
If the MSR researchers are here -- I'm curious what does it look like when
bordering hyperlapse with regular input? i.e., if there were a video
consisting of input frames at the beginning and the end, with a stretch of
hyperlapse in the middle, what does the transition look like? Does it need
much smoothing?

Also you probably saw this over the past week:
[http://jtsingh.com/index.php?route=information/information&i...](http://jtsingh.com/index.php?route=information/information&information_id=8)
(disregarding the politics of that) Whatever he's doing (I assume a lot of
manual work) it has a very similar effect and it has these beautiful
transitions between speeds.

Amazing work and the videos are stunning.

~~~
jkopf
Yes, I am here :-)

This would be possible. Although it would require providing some UI so the
user could specify which parts should be sped up.

I've seen the Pjong Yang video, it is beautiful work. It requires very careful
planning and shooting, and a lot of manual work to create such nice results.
We're trying to make this easier for CASUAL, but it's still FAR away from the
quality of professional hyperlapse.

~~~
vanderZwan
Great, now we can ask you questions! :)

Some of the videos demonstrate unusual "popping" effects and deformations when
standing still - especially notable in this video, top right, sixteen seconds
in:

[http://research.microsoft.com/en-
us/um/redmond/projects/hype...](http://research.microsoft.com/en-
us/um/redmond/projects/hyperlapse/supplementary/html/bike1_original_speed.html)

I understand how the extreme situation of climbing is a challenge, but what is
it about standing still that causes this? Do you have any thoughts on how you
might tackle this problem in future work? (although it appears you already
combine an amazing breadth of techniques, so I'm not sure how many options you
haven't looked at)

------
spindritf
The videos don't load for me (due to mixed content, I believe), so here they
are:

[https://www.youtube.com/watch?v=SOpwHaQnRSY](https://www.youtube.com/watch?v=SOpwHaQnRSY)

[https://www.youtube.com/watch?v=sA4Za3Hv6ng](https://www.youtube.com/watch?v=sA4Za3Hv6ng)

The hyperlapse of the climbing video looks like an FPS game from a decade ago
with texture refreshing as you get closer.

~~~
jkopf
Hmm... Did I not embed these correctly? The project page as well as the embed
links are plain [http://](http://) this is what it should be, no?

~~~
barrym
People using the HTTPS Everywhere extension will have the site URL as https,
but not the embedded videos.

------
jahmed
I walked around Boston once with some friends for 7 hours. When I remember it
I see it as the hyperlapse, not moment for moment or sped up. Super
interesting work.

~~~
ClassicFarris
I was thinking the same thing as I watched the video, that, this is probably
the closest to how most people remember long stretches of time.

------
steven2012
Okay please sign me up. I'm willing to pay hundreds of dollars for this
software. I have hundreds of gigabytes of time lapse that I've taken that is
just sitting there because of lack of ability to do something. I'd easily pay
$200+ for this software right now just so I can have those videos and free up
massive hard drive space.

~~~
NeilSmithline
I'm not sure that this software is what you want. It takes regular speed and
converts it to high speed. If you start with regular time lapse you'll need to
speed it up even more - maybe as much as 10x. Now you'll have super fast time
lapse.

~~~
steven2012
Actually, yes it's exactly what I want, I'm not sure why you would think I
misunderstood what the software did. I have hundreds of gigabytes of 1 sec
timelapse photos that could convert into pretty neat movies given what I saw.

~~~
zodiac
I think what NeilSmithline is trying to say is, the microsoft technology is
designed to convert regular 24fps videos to 2.4fps videos (10x time lapse). If
you feed it 1 sec timelapse photos (ie, video at 1fps), the output would be
0.1fps.

~~~
steven2012
I take 1 second time lapse photos and feed them at 60 frames per second for a
60x speed up. I'm not sure if you've made time lapse videos before, but it
wouldn't make sense to film something at 1 fps and then play it back at 1 fps.

~~~
zodiac
The frames per second I quote are all recorded frames per second (eg, for
every second of real-life time that passes, how many frames in the final video
were taken during that second?). Play-back rate is always 24fps (or some other
fixed constant).

What we're trying to say is, if you feed your video to the program, you're
going to get output that is sped-up 600x compared to real life. That's a
ridiculously high speedup.

------
moultano
I wonder how far you can get by using a "naive" timelapse of selecting frames
from the video, but being smarter about which frames you choose. Rather than
just choosing every nth frame, try to choose visually consistent frames by
making the intervals between the frames loose, then apply conventional
stabilization after the fact.

~~~
jdmichal
This was my initial thought about how they were doing this, but I don't think
it's as applicable as it would seem. At 10x speed up, that's still ~3 frames
from every second. I'd imagine a biker would spend at least a second turning
to look down an intersecting road before continuing through. So that would be
at least three frames where the perspective was heavily modified. It would
have to select for right before and right after the head turn and ignore
everything in between, which would probably create quite a jagged warp effect.

------
iamshs
Bloody amazing! Fantastic work! Release it. Release it soon. This is something
that everyone would want.

I see they have listed a Windows app coming. Is that Windows desktop app?

~~~
runeks
If I were them I would offer this as a web service: upload your video and have
it converted for $1.

~~~
rplnt
Someone above mentioned the processing took weeks. So.. I'm not sure that
would work.

~~~
sp332
Well, the _first_ version took weeks. I'm sure it will only get faster.

------
readerrrr
Great results but looks expensive. I wonder how many minutes of processing per
minute of video.

~~~
sambodanis
From the paper listed on the page it looks like it takes about 305 hours to
process a 10 minute video. The vast majority of that is during the "source
selection" phase which takes 1 minute per frame of video.

~~~
jkopf
We've already managed to dramatically reduce the processing time compared to
the numbers reported in the paper.

It'll still be slow (a couple hours for a 10 min video) but running on a singe
standard PC (with GPU).

~~~
sp332
It looks like after the frame-selection step, the rest of the process never
refers to the discarded frames. Is that right? Do you think making the frames
available for blending in the later steps would results in smoother blends?

------
adt2bt
This is so insanely cool. I plan to get a GoPro some day soon and will take it
on hikes in the Pacific NW. If I could turn my hikes into beautiful time-
lapses like these, I'd be blown away.

------
oh_sigh
Really great results. I wonder if this could be coupled with google street
view data to give a smoother time-lapse of a path.

~~~
oniTony
Right here — [http://hyperlapse.tllabs.io/](http://hyperlapse.tllabs.io/)

~~~
Zaheer
Just a note, this doesn't use the technology from the paper.

------
Lifescape
It appears as if the mountains are loading in the background (like in a video
game) as you get closer to them.

Awesome idea/execution!

~~~
0x0
Haha, I was just going to post that. It really gives off a mipmapping&LoD
game/progressive-loading effect. So weird.

~~~
hexasquid
And the running speed of the original Doom / Doom 2 games.

------
31reasons
Mind blowing results! Although the name Hyper-lapse doesn't really convey the
goal, it should be named Smooth-Lapse, because thats what its doing. Too much
hyper-x already.

~~~
Jack000
With the timelapse crowd, "hyperlapse" generally means a timelapse with a
moving camera (eg. reddit.com/r/timelapse) . In that sense they are using the
term correctly.

------
tehwebguy
This is incredible! If you want a good look at how it handles moving objects /
people check out the part at 2:15

[https://www.youtube.com/watch?v=SOpwHaQnRSY&t=2m15s](https://www.youtube.com/watch?v=SOpwHaQnRSY&t=2m15s)

------
itchmasterflex
Would it be possible to do something like this for audio? It would be
incredible to sample an hour-long album or mix in minutes.

~~~
limsup
time is kind of an essential component to music...

~~~
itchmasterflex
Time is supposed to be essential to video as well.

This video [http://vimeo.com/13669078](http://vimeo.com/13669078) has time-
lapsed audio. I wonder what it would sound like using the hyperlapse effect.

~~~
socksy
This is intriguing. What would you need to model as a the continuous input to
try and get from the timelapse? With the imagery, a model was made of the 3D
scene in which was then used for the 2D final output.

I'm having difficulty imagining what this more in-depth model would represent,
and how you'd either strategically take the clips to "paint" this model.

------
photojosh
I would use this for sure. I do timelapses of runs I do and set as challenges
for our social running group. The source is a head-mounted GoPro. The problem
with them is that a straight forward pick-every-nth-frame gives a motion-
sickness inducing video, as well as blurry. If you could extract the frame
from the top of each stride when the camera is most steady, I would imagine it
would be very much more watchable.

------
washedup
This is incredible. By watching the hyper lapse versions of the mountain
climbing, you can clearly see which path is taken, are able to get glimpses of
whatever paths are available. This would be a huge advantage for people
learning how to rock climb. I can image that a similar situation would occur
for many other activities. Great work!

------
jelveh
left side video was shot here:
[https://www.google.com/maps/place/Altona+Town+Hall/@53.54767...](https://www.google.com/maps/place/Altona+Town+Hall/@53.547678,9.935551,3a,75y,208.15h,90t/data=!3m4!1e1!3m2!1srIxKeAhLQAZTOOF3zZg8Bw!2e0!4m2!3m1!1s0x47b185872fd5a80b:0x50ce599e7d52265b)

~~~
junto
Was the other longer video also from Hamburg? For a moment I thought it was
Berlin.

------
sabalaba
This is great. I have weeks of footage from a camera that I wear around and
would love to use that video to make a hyperlapse. I would also be interested
in seeing how well this does with photos taken every few seconds as opposed to
video. Although, after reading the paper, it looks like there would be a lot
of optimization that would need to happen to make it more efficient. (Their
original implementation took a few hours on a cluster.) Luckily, as they
stated in the technical video, they haven't tried to do anything more than a
proof of concept; so there is plenty of room to optimize. I'd be interested to
see how well a single-machine OpenCL or CUDA implementation does compared to
the CPU clusters they were using in the paper.

------
closetnerd
That honestly has fantastic results.

------
Yuioup
I'm surprised Microsoft never releases anything cool like that to their app
store.

~~~
burkaman
>We are working hard on making our Hyperlapse algorithm available as a Windows
app. Stay tuned!

~~~
Yuioup
Ah cool. Over the years Microsoft Research has also released demos of other
cool stuff, like recreating a 3d view using multiple photos from different
angles. How about those projects?

~~~
lucb1e
> recreating a 3d view using multiple photos from different angles

I've been waiting for that for years. Searched for it a few months ago, still
nothing.

~~~
grkvlt
Er, [http://photosynth.net/](http://photosynth.net/)

~~~
lucb1e
This is for panoramas, not 3d objects right? I meant merging photos of an
object from different angles.

Edit: It's not even working. Photographed something from different angles,
synth it, and it appears on their website as a slideshow. Like a normal jquery
slideshow except you need Silverlight®.

~~~
burkaman
I'm not sure why the site isn't working, but the software is definitely for 3D
objects, not just panoramas. Maybe your photos didn't have enough in common
for it to work?

------
pwenzel
The city demos remind me of the Blues Brothers' car chase scene in downtown
Chicago.
[https://www.youtube.com/watch?v=LMagP52BWG8](https://www.youtube.com/watch?v=LMagP52BWG8)

------
rumham
Incredible results. I'd kill to try this with Oculus Rift.

~~~
31reasons
What about motion sickness. I can't imagine trying lapse videos in the Rift
without puking in one minute.

------
bitL
Very nice! Is there software available I can use, or do I have to implement
the algorithms from the paper myself?

I make a lot of 4K hyperlapse movies, it is tedious as AfterEffect's warp
stabilizer is useful only in a small fraction of cases, Deshaker is more
consistent but also not perfect, and the only option in the end is multi-pass
manual tracking and stabilizing which is very time consuming and tricky for
long panning shots.

~~~
adijr
according to the website: "We are working hard on making our Hyperlapse
algorithm available as a Windows app. Stay tuned!"

------
aceperry
Pretty cool that they've included the raw video. Everyone can use that to
compare their alternative method to the microsoft researchers' results.

------
bsimpson
As others have commented, the videos look great, and much closer to how people
remember journeys. However, there appear to be some image persistence problems
(many street poles simply dissolve as they get closer to the camera).

I'm curious to see what happens if they insert more action-packed footage. An
MTB course with trees, switchbacks, and jumps would be an interesting stress
test of this technique.

~~~
sp332
They do frame selection first, then create a 3D scene from the fewer frames.
So that light pole might not have been in enough of the chosen frames to get a
good 3D model of it. And the first stage has a pretty low-res 3D model because
the number of points just gets crazy, so the light pole probably wouldn't have
been modeled anyway.

------
NicoJuicy
What's the best alternative for this? I'm doing a GoPro video while cycling
with some friends... And this would be insanely usefull

------
the_cat_kittles
be sure to watch the technical exegesis at the bottom, its almost more amazing

------
jalopy
Wow please release this either standalone or even better as a feature of some
MS movie production tool. iMovie needs some competition.

------
issa
I want to put something more meaningful into this comment, but all I can think
to say is that this is really amazing. Well done!

~~~
RankingMember
Pretty much my thoughts exactly. It's also nice to see Microsoft get a little
press about something innovative/cool. :P

------
bellerocky
I don't understand how computers figure out 3D from single camera video
without parallax. How do they do that?

~~~
alok-g
You track points in the scene across frames. This helps infer the 3D position
though getting good accuracy is hard. Basically parallax gets created in time
as the camera moves.

I recall those childhood days when we could not explain why moon appeared to
come along as our car moved. :-) Those differences in apparent speed betrays
something about the distance of the object/point under consideration.

------
hrjet
I downloaded the .mp4 file and watched at half-speed. It looks great at half-
speed too, and much more realistic. I wonder why they couldn't have slowed it
down a notch. Perhaps, as a research result, they are just staying true to
their algorithm's output frame rate.

------
rbanffy
Couldn't it be done with a wider angle lens, a better imaging sensor and
conventional image stabilization techniques? If such captures become
commonplace, it's easy to imagine capturing with a wider field of view so that
the stabilizer would not be so overtaxed.

------
ntenenz
Szeliski also has written a comprehensive Computer Vision book for those
looking to learn the field. It's a fairly good one (and available for free
online - [http://szeliski.org/Book/](http://szeliski.org/Book/)).

------
crahrah
I'm thinking of trying to recreate this - does anyone know if this is covered
by a patent?

------
GhotiFish
awww. I always have mixed feeling about a new microsoft funded tech. Something
really really cool that I will never get to use/exploit/play with/anything.

The technical video breaks down some of the techniques they used. Global match
graph is particularly interesting. This technique alone could lead to a big
improvement in timelapses, by trying to select consistent changes between
frames.

[http://cg.cs.uni-
bonn.de/aigaion2root/attachments/FastSimila...](http://cg.cs.uni-
bonn.de/aigaion2root/attachments/FastSimilaritySearch.pdf) <\- maybe this?

------
tumes
This looks like something Michel Gondry would've figured out manually 20 years
ago with an 8mm camera and a paper cup or something.

That being said, it really does look amazing!

------
bobwaycott
One of the most impressive things I can recall seeing from Microsoft in years.
Would absolutely love to have this as a cloud service or desktop app.

------
sjtrny
Does anyone know how the match graph is created? (The part that is used to
identify redundant frames). The paper barely mentions it.

------
jcubic
Now I wait for github repo with open source library or app that will create
this videos.

~~~
shirman
same here

------
michaelmachine
Woah, very cool. I would love to see this applied to a SCUBA diving video.

------
joseman
Very excited about this being released as a product. HURRY UP!

------
zobzu
wow i wanted this for a while

now to implement it open source ;)

------
lukasm
Is this Hamburg? "getrankemarkt" :)

~~~
gorekee
Yes, it's altona nord.

------
Netcob
A GoPro timelapse is also called a prolapse.

~~~
scentoni
How unfortunate.
[http://en.wikipedia.org/wiki/Prolapse](http://en.wikipedia.org/wiki/Prolapse)

------
unphasable
wow this is incredible!

------
suchetchachra
Excellent work!

------
gcb0
i would pay to have that on a video player fast forward feature.

------
manos_p
still sucks

------
anvarik
why paper is 35MB?

------
jaequery
U

------
EvanL
Cool technology, applications?

------
tibbon
I'll be the immature one and say I'm curious how funny porn would turn out
with this speeding it up by 10x. They don't show much of how it deals with
people and I imagine the results would be terribly funny looking- but perhaps
awesome.

Also, I will pay $$$ for this to use with my motorcycle footage from GoPros.

~~~
m_mueller
My guess is it doesn't work well with footage with cuts in it, it's probably
meant for continuous shots (there are a few long ones in Hugo and Gravity for
example, so this might be interesting input material). From how the climbers
appear, porn would probably decompose into static images.

------
tannerc
Stunning effect, can anyone help me see the practical or entertainment-value
use?

I'm also curious if anyone else got motion sickness while watching the video.

~~~
maxmcd
If I could just take out my iphone and lazily records a skateboard ride down
manhattan, and then get this result, that would be incredibly compelling.

For a slightly more practical use this could be a tool to give people previews
of hiking trails, tours, or routes they're about to take.

------
l33tbro
Great tech. But it really just looks like a steady-cam sped up x10. I get the
technical brilliance, but visually it's not terribly innovative.

------
melling
Now we need a little facial recognition so you can scan to where you meet your
tagged friends... of course, there are other surveillance opportunities too.

------
philip1209
The server supports HTTPS, but the videos are improperly embedded resulting in
mixed-content errors. This is disappointing.

~~~
apu
Seriously, this is what you're going to talk about? Nothing about the
technology? And it's "disappointing"?

------
rasz_pl
Nice results, but missing forest for the trees.

One of the by-products of this algorithm is fully textured 3d model
representing filmed environment. Offering that as pure data dump, or even a
manual process allowing user to control camera would be as valuable as fully
automatic one-off timelapse no one ever watches (except maybe your granny).

What sounds better - a video tour of a house, or a 3D model of a house you can
traverse however you like?

I wonder if 3 letter agencies have better structure from motion
implementations a la "Enemy of the State" (Isnt it sad that this film turned
out to be a documentary?). I suspect something like a 3d reconstruction of
Boston Marathon (FBI did collect all video footage of the event) would be very
helpful to the investigation.

~~~
baddox
Generating a 3D model of an environment from the output of a moving camera has
been done. There is obviously a lot of improvement to be done in that
department, and those projects are neat, but I think it's appropriate for this
project to focus on what it adds to the scene, which is camera path smoothing.

~~~
pbhjpbhj
Indeed it has, photosynth - I blogged about it before,
[http://alicious.com/making-3d-models-from-
photos/](http://alicious.com/making-3d-models-from-photos/),
[http://photosynth.net/preview/view/6de2eb38-4e7e-424b-94a3-9...](http://photosynth.net/preview/view/6de2eb38-4e7e-424b-94a3-93a6f35712ef)
\- which came from Microsoft Research. It looks like this is "just" the video
version of what they were doing before with photosynth walkthroughs. Very
impressive.

------
boyaka
Video stabilization + more FPS / slower rate than the "every 10 frames
timelapse" \+ feel good inspirational music = this

I would guess that I could upload a shaky video to youtube to get it smoothed
out, download it, and speed it up with similar to their rate and get similar
results. The timelapse that they show that is so much worse uses way less
frames of the raw footage (every 10th frame?) and goes way faster than their
"hyperlapse". It isn't a fair comparison.

~~~
Renaud
>I would guess that I could upload a shaky video to youtube to get it smoothed
out, download it, and speed it up with similar to their rate and get similar
results.

No you certainly wouldn't. Watch the technical video at the bottom of the
page. It will explain why this is not trivial to do and why standard
stabilisation technologies aren't useful to smooth out time lapses.

~~~
boyaka
Well, I admit that I was pretty ignorant about the work being done on this
project in regards to time-lapsed video. I guess I could add to my previous
statement that they also cut out irrelevant frames (parts of the video that
aren't in the camera path). I don't think this would be THAT difficult to do
manually, but I admit that the technical video showing how they were able to
graph/visualize the irrelevant frames is pretty cool, and the interesting
resulting effects people are discussing in this thread (disappearing/appearing
objects, the video game loading effects) are amusing.

I never said that it was trivial, just that similar stuff has already been
done and made a "standard stabilization technology", automatically and easily
just by uploading to youtube. It seems that youtube's techniques aren't
necessarily completely different: there's a screenshot of an article from
Google in this video [1] called "Auto-directed Video Stabilization With Robust
L1 Optimal Camera Paths". However, I do appreciate and shouldn't disrespect
the specialized work being done for time-lapsed videos. My apologies.

[1]
[https://www.youtube.com/watch?v=BgAdeuxkUyY](https://www.youtube.com/watch?v=BgAdeuxkUyY)

