
Denser 3D Point Clouds in OpenSfM - yubin
http://blog.mapillary.com/update/2016/10/31/denser-3d-point-clouds-in-opensfm.html
======
jsch
COLMAP is an open source structure-from-motion and multi-view stereo software
and is hopefully useful for people here as well:
[https://colmap.github.io/](https://colmap.github.io/),
[http://people.inf.ethz.ch/jschoenb/colmap/](http://people.inf.ethz.ch/jschoenb/colmap/)

------
deckar01
I discovered VisualSfM this weekend after trying Adobe ReCap 360. The photo
reconstruction functionality of ReCap is cloud based, which was a blocker for
me, because I have a beefy graphics rig and want to script it into a pipeline.

I am curious how OpenSfM compares to VisualSfM and if they are related in any
way.

Edit: The VisualSfM license is only open for non-commercial uses. OpenSfM is
BSD licensed, so I can safely build my business on top of it.

~~~
yubin
Both VisualSfM and OpenSfM use similar methodology (incremental structure from
motion) to reconstruct 3D scene points and camera positions from photos.

And yes, you can build your own tools (even commercially) with OpenSfM.

~~~
deckar01
I am curious how the quality compares and if they have common library
dependencies.

------
efkv
For those who have not tried it, OpenSfM is an amazing tool. Can't wait to
play with dense reconstructions. Congrats mapillary team, I know this has been
a long time coming!

------
yummybear
I wonder if you wouldn't get cleaner results if you eliminated points that
don't have an adjacent point nearer than a certain distance?

~~~
yubin
Yes. We can apply different heuristics to remove outlier points. One
interesting way to do it will be to train a neural network to refine the dense
point cloud.

------
Pica_soO
Do they use the same NeuralNets that are used to improve videos on these
pictures to reconstruct the details?

------
data0815
How well would this work in a situation with artificial (moving) lighting? I
have seen reconstructions for wrecks underwater, and we are interested in
doing something similar.

~~~
rasmi
It is possible to correct for all sorts of different lighting conditions. See
here:
[http://grail.cs.washington.edu/projects/timelapse/](http://grail.cs.washington.edu/projects/timelapse/)

------
BHSPitMonkey
What techniques are available to transform point clouds like these into closed
polygonal meshes (like the ones you see in Google Earth where 3D imagery is
available)?

------
MichailP
Is it possible to use images from say Google Street View to reconstruct 3D
city model with this tool? Maybe it could fall into fair use for small part of
city...

~~~
toomuchtodo
You'll want to use the OpenStreetView dataset for that, which has a much more
liberal license for collected images than Mapillary. (OSV is a direct
competitor to Mapillary, with OSV being sponsored by Navteq)

[http://openstreetview.com/terms/](http://openstreetview.com/terms/)

You might also consider mining Flickr by geotag, constraining to CC licenses
that allow for use.

~~~
maxerickson
Or even just directly capture geotagged photos. That's going to be the best
way to ensure good coverage of the target area.

~~~
toomuchtodo
What would the capture source be if not an existing online repository?

~~~
maxerickson
A camera.

That's where Mapillary and OSV get their imagery from, people that decide to
go out and take pictures.

An additional advantage is that you then have full rights to the imagery (so
for example, you can upload them to both services).

~~~
toomuchtodo
That'd work of course! Bit more of a schlep to build out your processing
pipeline to ingest those images, store them, and then render them with
OpenSfM, but could definitely be done.

~~~
maxerickson
Oh yeah, it'd be quite some work, especially if small isn't small. But I think
at the moment, neither OpenStreetView nor Mapillary can be relied on for
coverage, so the shortcut of using images gathered by others isn't widely
available.

------
dharma1
very cool. that's quite dense.

So the segmentation convnet is applied on the 2d image pixels before SfM,
rather than on the point cloud data? I've been looking for neural net semantic
segmentation for 3D point cloud data

~~~
yubin
Exactly, the convnet is applied on 2d pixels instead of on 3D point clouds. If
one have labelled 3D points (e.g. if a point is part of a car or not), one can
train a network for recognition directly on 3D points.

~~~
dharma1
Nice. I wonder if this could be used for robot navigation. The segmented point
clouds especially are useful.

Been looking forward to a DIY Roomba project using point clouds and neural
nets for navigation

------
eriknstr
Never heard of OpenSfM before but have been looking for something like it.

~~~
spacemanmatt
I have personal and professional projects that involve mundane data feeding
and service-building around this core capability. This totally fills a niche
that's been on my tech radar for a while.

------
zump
So why can't this be used instead of LIDAR for Self-Driving cars?

Experts?

------
zubairq
Pretty amazing stuff!!

------
washwash
great

