
Fused Video Stabilization on the Pixel 2 and Pixel 2 XL - stablemap
https://research.googleblog.com/2017/11/fused-video-stabilization-on-pixel-2.html
======
codinghorror
This -- is super freakin' cool. Real value add from Google, considering
everyone is on the same very mediocre Qualcomm Snapdragon 835 platform.
(Please PLEASE for the love of God, Google, build your own mobile SoCs so
someone can remotely catch up to Apple's 3 year lead in mobile performance
already)

Credible too because Pixel 2 had such stellar picture quality in the reviews.

~~~
samfisher83
Chip making isn't magic. The more transistors you have the more things you can
do. For example you can have more cache, more decoding units, more execution
units etc. All this increase in complexity has costs which other phone
manufacturer may or not want to pay for. Apple has a billion more transistors
to work with. Will most consumers even notice the difference? will they pay
more? Apple has a high asp so they can do it. How many other manfuctuers can
do that?

~~~
tertius
> Will most consumers even notice the difference?

Well, it allows them to advance UI instead of just making it faster.

------
BugsJustFindMe
> _Fused Video Stabilization on the Pixel 2 and Pixel 2 XL_

Aside from gatekeeping, what does any of this have to do with the Pixel 2? Why
is the headline not "Fused video stabilization for any Android device with a
camera"?

I mean, Microsoft published inertial image deblurring using entirely
unintegrated devices in 2010 ([https://www.microsoft.com/en-
us/research/publication/image-d...](https://www.microsoft.com/en-
us/research/publication/image-deblurring-using-inertial-measurement-
sensors/)), though I swear I remember seeing a poster from them earlier than
that.

(Also, I can honestly say that I've never seen a video recorded on a phone
that had motion blur or shutter wobble anywhere near as bad as what they show
in their examples, so those examples seem contrived, but obviously it's great
to have an algorithm that can handle exceptionally bad cases.)

~~~
TheAdamist
Needs output from what the ois is correcting too for this to work, which I
don't think most ois systems output.

~~~
BugsJustFindMe
See, I don't believe that that is true. There might be some additional
information gained there, but it's not _needed_, as evidence by a decade of
publications on the subject of correcting motion artifacts.

~~~
CarVac
Without knowing where the lens is in relation to the sensor, you can't correct
for the time-varying perspective distortion.

Perhaps the decade of publications have never involved optical image
stabilization in the loop.

What you _could_ have, however, is a simulation of where the OIS moves the
optical axis of the lens to, when given the gyroscope inputs. In control
theory terms, an open loop observer. But that would need to be calibrated to
each model of phone.

~~~
BugsJustFindMe
> _But that would need to be calibrated to each model of phone._

We needn't suddenly forget that every android phone asks you to wave it around
in figure eights to calibrate the compass (and then still only seems to be
accurate to within +/\- 180 degrees). Making every video you take come out
better on every Android device with sufficient CPU power after a few seconds
of recording would be significantly preferable to "Only works on our devices,
neener neener." To me, obviously, maybe not to Google.

~~~
cromwellian
That kind of calibration isn't the same as factory calibration of IMU, camera,
OIS systems against some known ground truth.

------
sandGorgon
Way too many things that are Pixel only. Does this mean that the Android open
source era is over ?

------
demarq
I knew there was quite a bit of processing going on in flagship cameras but I
never expected this level. Really impressed!

------
exodust
Cropping is still happening, and it can gobble up your content if you're not
careful.

Even in their demo video of guy jumping down sand dunes, we see the
destructive price of 'fused cropping stabilisation'. The most important shot
of guy jumping misses the moment his feet hit sand at 2 sec in when he does
his biggest leap.

It's details like these, and the clouds of sand he kicks up behind him which
make an interesting shot. I like my focal length untouched, thanks anyway,
optical stabilisation all the way for me.

Here's the sand dune video.
[https://youtu.be/kaO7Gs-F2vA](https://youtu.be/kaO7Gs-F2vA)

I wonder if there's any manual control over crop factor in the Pixel settings.

------
Klathmon
That motorcycle video near the end of the article is simply amazing.

~~~
rasz
so were the demos of Youtube video stabilization, in reality Youtube EIS is
absolute garbage.

~~~
boundlessdreamz
It seems to work quite well. See
[https://youtu.be/0Mp5Y3UTWH8?t=93](https://youtu.be/0Mp5Y3UTWH8?t=93)

------
isoprophlex
I still think most software video stabilization looks terrible and unnatural.

What's wrong with focus breathing and perspective shifts?

~~~
readams
The terribleness and unnaturalness is precisely the problem they are trying to
solve with this solution. The results look very impressive compared to
previous solutions.

------
vizzah
iPhone user looking with envy. Is anything of a similar magnitude available in
a standalone software package? For already recorded videos?

~~~
CharlesW
> _iPhone user looking with envy._

I'm not sure FOMO is warranted, since Apple's so-called 'cinematic video
stabilization' (introduced in the iPhone 6 days) is quite good. Additionally,
the iPhone 6 Plus, 7 Plus, and both iPhone 8 models have optical image
stabilization.

~~~
xster
I think the whole point of the article is that simple OIS isn't enough.

------
mozumder
Two things left to improve for mobile phone video that would massively
increase image quality: increased dynamic range and higher bit rate.

Increasing dynamic range in going to be tough. The only way to to do that
would be to make the pixel's electron well bigger, but phone cameras have
limited pixel size. Image sensors will probably have to provide a separate
stacked electron well capacitor below each pixel, instead of using the
photosensor PN junction.

Also, the industry really needs a standard raw video format that allows for
high-quality post exposure correction and white-balance.

------
microcolonel
Something I'd like to see is a square sensor, so you can take horizontal video
in portrait if you forget.

