Part of the reason I worked on this in my spare time (my real job at Google is something else) was to get it integrated in libmv (http://code.google.com/p/libmv) and subsequently integrated into Blender. This opens the door to many sophisticated tracking features, like autocalibration, rolling shutter, multi-shot solves, tripod solves, planar tracking, and more.
EDIT: Changed link title to make it more provocative
If you're snapping track data to roads, how do you deal with areas where you have no existing data? Do you take into account aerial imagery?
I took some notes during the StreetView Q&A on reddit two years ago: http://blog.est.im/archives/554
I was looking forward to integration of motion tracking into Blender and had a small play with the feature when released. Now I just need to make the time to play some more...
Note that Blender's next open movie, http://mango.blender.org, is live-action done with Blender's tracking tools. Ceres is going to be used for that as well.
Note this supposition is based almost entirely on doing the first 3 weeks of the Udacity course CS373: Programming a Robotic Car, but it seems to fit quite neatly with the stuff they talk about i.e. a car/robot moving, guessing how far it's moved, taking sensor snapshots of its environment then cleverly combining the data to figure out accurately where it is.
Off-topic: scrubbing back and forth in the blog video (to get a better idea of what was happening) seemed incredibly smooth for me, is this a side effect of Youtube/Firefox using HTML5 video? (edit: trying it out with Flash, I get a fancier, but less useful pop-up when I try to scrub, a tech limitation of Flash or just a design decision?)
* not meaning to underestimate the hard work and genius that underlies this technology, but the same concept is used in such everyday items as Wii remotes and suchlike.
I mean, when you have a loop (say, around the manhattan grid), then I can understand how everything works when you have one hard(er) constraint such as, "the two farthest points visited are 2nd street at avenue C and 112th street at 12th avenue" - and get everything else aligned accordingly.
But if no external geometry constraint is involved (roads being the most abundant, but not the only type of external geometry constraint), then this reader has been unsuccessful with the exercise.
It's really surprising how broadly applicable non-linear least squares is.
This is extremely cool. The documentation seems to be a WIP on this, can anyone comment on what kind of cost functions are supported by automatic differentiation?
EDIT: it would be also interesting to see how this compares to Theano , which also does symbolic differentiation and can JIT to native code and GPU. It is a largely more generic framework, but I'm not sure how well it can handle large sparse problems.
What happens if I try to use an if-then or a for loop depending on one of the variables?
In particular, search for "SnavelyReprojectionError". In this case there are no loops or if statements, but they work fine too. The only issue with branches is that you can make the cost function discontiguous, which can cause issues.
The way this works is that you write your cost functions templated on T. For pure-cost evaluation, this is a double. For cost and jacobian, T is replaced with the "Jet" object, found in jet.h. However, you never see this; Ceres does the substitution for you.
There are some caveats; for example, sqrt(0) doesn't work on autodiff objects since the derivative is undefined. You have to use a taylor approximation or similar if the argument to sqrt is exactly zero.
If you want to see some more complicated code that runs with autodiff, check out the included rotation.h header. We use that internally for 3D reconstruction type cost functions:
So conditionals are not a problem. Loops I would stay away from, since they would destroy the inlining. But if you have an example of a dynamic loop, we would like to hear about it.
I think it can be used as a backend in Ceres to push computations to the GPU, but as it stands it is a different category of software than Ceres.
While we do support numeric differentiation, we don't suggest you use it. What we have is automatic differentiation, which is a technique to take exact derivatives without resorting to symbolic differentation.
Check out he jet.h file which is the core implementation of this:
The header comment has a nice description of autodiff.
In summary, Ceres supports three ways to compute derivatives:
(1) Automatic differentiation (easiest, fastest, most accurate)
(2) Numeric differentiation (easy, but worse convergence and hazardous)
(3) User-supplied jacobian (use a pen and paper then implement the jacobian for your residual manually)
(4) Complex-step differentiation
The technique is clever: they add an finite difference along the imaginary axis, and this turns out to give a more accurate derivative than normal forward differences.
We don't support this in Ceres and suggest autodiff instead.
What I meant to say was, "If it's all in one C++ project, it's not symbolic differentiation" and I took the liberty of changing "not symbolic" to "numeric."
Having now read your posted code, I kind of feel like a 17th century naturalist trying to classify a platypus. Clearly this technique is not symbolic differentiation, since it can only produce numerical and not symbolic results (although I suppose the Jet template could handle an "expression" type as its first argument - have you actually tried this?), and it's not really numerical in the normal sense because there's no h parameter that changes the accuracy as it varies.
As an aside, "autodmatic ifferentiation" is a terrible name, in the sense that it doesn't convey any real information about the technique, except maybe that it's being done by a computer and I knew that already. It might still be a good marketing term (like how Richard Bellman named his technique "Dynamic Programming" because it's impossible to call something "dynamic" in a perjorative sense, although there really isn't anything intrinsically dynamic, and in fact you commonly end up solving "static dynamic programming" problems).
Numeric differentiation has a specific meaning - which is computing derivatives via finite differencing.
There's three ways to compute derivatives; for whatever reason most people only know about symbolic (take the derivative by hand, implement it in C++) and numeric (implement your cost function, do finite differences on each parameter to determine the gradient) differentiation. Automatic differentiation is a totally different way to take derivatives. The Wikipedia article about it is fairly good.
We didn't pick the name.
For example, in the subproblem of only one-way streets that we can assume to be composed of one or more segments of straight lines per street, we take a simple street-function $\phi_i(x,y)$ corresponding to street i to be the norm of the projection of <x,y> onto that street. Furthermore we also add into the system some "smoothing-function" to ensure that the overall shape of the final path is doable, for example constraining the distance between successive points. Next, we solve the argmin of the norm equation for each point so that each point is now moved to some linear combination of the streets, and truncate all but the most significant street basis, and rerun until we get to some acceptable tolerance.
Its a bit like Google being the biggest supporter of Firefox, even though they are also a competitor (Chrome.) Though Google supporting OSM is more like Microsoft supporting OpenOffice, since Google has such a large monopoly in maps right now.