

Google open sources key piece of Street View technology - mierle
http://google-opensource.blogspot.com/2012/05/introducing-ceres-solver-nonlinear.html

======
mierle
This is a key component of Street View. We're really happy to have released
this. When we started writing Ceres, there was nothing available like it. It's
state of the art; a real contribution to computer vision and other fields
which need large scale nonlinear least squares.

Part of the reason I worked on this in my spare time (my real job at Google is
something else) was to get it integrated in libmv
(<http://code.google.com/p/libmv>) and subsequently integrated into Blender.
This opens the door to many sophisticated tracking features, like
autocalibration, rolling shutter, multi-shot solves, tripod solves, planar
tracking, and more.

EDIT: Changed link title to make it more provocative

~~~
yellowbkpk
When you say "a key component of Street View", which part is it a key
component of? Snapping vehicle track data to existing roads? 3D model
computation for scene-to-scene jumps? Both?

If you're snapping track data to roads, how do you deal with areas where you
have no existing data? Do you take into account aerial imagery?

</openstreetmap-nerd jealous="true">

~~~
est
Google Street View was filmed at 30fps video format. You can generate a 2D
route map directly from it without GPS or when GPS is not available.

I took some notes during the StreetView Q&A on reddit two years ago:
<http://blog.est.im/archives/554>

------
sparky
Awesome! I just filed a few bugs with patches that let it build out-of-the-box
on Ubuntu. There didn't seem to be much in this space outside of levmar (
<http://www.ics.forth.gr/~lourakis/levmar/> ); levmar is great for what it
does, but some friendly competition is most welcome from my perspective.

It's really surprising how broadly applicable non-linear least squares is.

~~~
mierle
Working on integrating fixes now...

~~~
mierle
Done!

~~~
EvilTerran
Wow. Released, then community-sourced patches offered and integrated, all in
ten hours. Sometimes open source makes me feel all warm & fuzzy inside. :)

------
ot
> Automatic differentiation

This is extremely cool. The documentation seems to be a WIP on this, can
anyone comment on what kind of cost functions are supported by automatic
differentiation?

EDIT: it would be also interesting to see how this compares to Theano [1],
which also does symbolic differentiation and can JIT to native code and GPU.
It is a largely more generic framework, but I'm not sure how well it can
handle large sparse problems.

[1] <http://deeplearning.net/software/theano/>

~~~
sandwichmaker
You can define define arbitrary functions. You are only limited by the
operator overloads defined in jet.h. Let us know if you think something is
missing.

~~~
ot
I guess you mean arbitrary _expressions_ , just using arithmetic operations
and the overloaded functions (I see that abs, log, sin, cos, ... are
supported).

What happens if I try to use an if-then or a for loop depending on one of the
variables?

~~~
mierle
It's not just expressions; it's any computation you normally would do with
doubles. Take a look at the simple bundle adjustment example:

[http://code.google.com/p/ceres-
solver/source/browse/examples...](http://code.google.com/p/ceres-
solver/source/browse/examples/simple_bundle_adjuster.cc)

In particular, search for "SnavelyReprojectionError". In this case there are
no loops or if statements, but they work fine too. The only issue with
branches is that you can make the cost function discontiguous, which can cause
issues.

The way this works is that you write your cost functions templated on T. For
pure-cost evaluation, this is a double. For cost and jacobian, T is replaced
with the "Jet" object, found in jet.h. However, you never see this; Ceres does
the substitution for you.

There are some caveats; for example, sqrt(0) doesn't work on autodiff objects
since the derivative is undefined. You have to use a taylor approximation or
similar if the argument to sqrt is exactly zero.

If you want to see some more complicated code that runs with autodiff, check
out the included rotation.h header. We use that internally for 3D
reconstruction type cost functions:

[http://code.google.com/p/ceres-
solver/source/browse/include/...](http://code.google.com/p/ceres-
solver/source/browse/include/ceres/rotation.h)

------
cft
That's no Instagram, no Facebook, this is technology, not Hollywoodisation of
Silicon Valley

------
jacquesm
Neat, what a pity that real hacker news gets under-appreciated!

------
jetsnoc
Thank you for your contribution. It's great to see a giant, who's gained so
much from open source contribute back in a significant and meaningful way! For
those interested it appears it was open sourced under the "New BSD license."

------
strebler
What exactly is the operation being done in the video? My best guess is a
noisy set of GPS coordinates / panoramas being repositioned via bundle
adjustment on the street view panoramic captures. Or something.

~~~
sandwichmaker
In the video, no vision/bundle adjustment is used, just GPS, IMU and other
sensors.

~~~
leegao
I think I can sort of see what's going on, we're trying to fit the data of
some GPS position taken from a walk around the park or something similar, and
fit it to some linear combination of basis of some "street functions" to
minimize its two-norm, where the "street function" corresponding to each
street for some point (x,y) is some measure of how far away from the street
that point is.

For example, in the subproblem of only one-way streets that we can assume to
be composed of one or more segments of straight lines per street, we take a
simple street-function $\phi_i(x,y)$ corresponding to street i to be the norm
of the projection of <x,y> onto that street. Furthermore we also add into the
system some "smoothing-function" to ensure that the overall shape of the final
path is doable, for example constraining the distance between successive
points. Next, we solve the argmin of the norm equation for each point so that
each point is now moved to some linear combination of the streets, and
truncate all but the most significant street basis, and rerun until we get to
some acceptable tolerance.

Maybe?

------
JoeCamel
Johnny Lee has written a short post about Ceres:
[http://procrastineering.blogspot.se/2012/05/today-sameer-
aga...](http://procrastineering.blogspot.se/2012/05/today-sameer-agarwal-and-
keir-mierle-as.html) "If I find the time, I might try to post some tutorials
on using Ceres. Because I believe this is one of the most powerful tools in
modern engineering, and no one ever taught it to me in undergrad or high
school. It's like the difference between doing long division by hand and then
being handed a calculator."

------
pkh80
It'd be great if this is just the start of a trend where Google supports open
source mapping tech that would make projects like OSM more viable (for reasons
why its not currently viable see my other posts.)

Its a bit like Google being the biggest supporter of Firefox, even though they
are also a competitor (Chrome.) Though Google supporting OSM is more like
Microsoft supporting OpenOffice, since Google has such a large monopoly in
maps right now.

------
eliben
From a cursory look - that's a nicely organized, formatted and documented
piece of code. Well done.

------
slig
In the video they figure out the road inside a tunnel, with no GPS data. How
does it do that?

~~~
mierle
Fusing sensor data (IMU, etc). You can formulate this as a (extremely) giant
sparse least squares problem.

------
jlapenna
The video in the link is awesome. So that's how Google figures out roads and
stuff!

------
joelthelion
This is __much more __than a key piece of Street View. It's an awesome
building block for thousands of very diverse AI applications. Great
contribution!

------
rmc
Google has often contributed code to the open source world (like this).
However they don't AFAIK release any data. Google drove cars around the world
photographing everything. But they claim copyright on it all. If they allowed
derivative works of it, then Open Street Map could bloom and benefit
massively.

~~~
rmc
Downvotes because I point out how Google can donate data, not just code? ☹

------
fgcallari
Yeah!!

~~~
mierle
Took long enough but we finally made it!

------
julienr
Good job !

