
Atlas: End-to-End 3D Scene Reconstruction from Posed Images - taylored
https://github.com/magicleap/atlas
======
cs702
From the folks at Magic Leap. It looks remarkably good to me.

The video at
[https://www.youtube.com/watch?v=9NOPcOGV6nU&feature=youtu.be](https://www.youtube.com/watch?v=9NOPcOGV6nU&feature=youtu.be)
is worth watching, especially the parts showing how the model gradually
constructs and improves a labeled 3D mesh of a live room as it is fed more
visual data by walking around the room.

\--

On a related note, Magic Leap has been trying to find a buyer for the business
for several months now:

[https://www.roadtovr.com/report-magic-leap-buyer-
sale/](https://www.roadtovr.com/report-magic-leap-buyer-sale/)

[https://www.bloomberg.com/news/articles/2020-03-11/augmented...](https://www.bloomberg.com/news/articles/2020-03-11/augmented-
reality-startup-magic-leap-is-said-to-explore-a-sale)

~~~
caiobegotti
I have no experience in this field at all and they note on the video that the
sequence shown was not realtime but I wonder how far we're from having
something like this running in realtime or how "realtime" it could be given
fancy hardware to be used in the wild?

~~~
taylored
Without much optimization, it can run at ~14fps on a NVidia TiTan RTX

------
kanobo
On a tangential thought, it's interesting to me that a company (magicleap)
that has raised several billion dollars generates so little value compared to
other companies its size that this is the most notable output from them in a
year and I thought it was a phd project until I looked at the project owner.
Anyways, it's a very interesting project and thanks for sharing.

~~~
rayuela
Yeah I have to agree. If this were a PhD thesis it would certainly deserve
some praise, but given that this is the most exciting thing to come out of
magic leap in years just barely puts them on par with SOTA...well I would be
pretty pissed if I was an investor in them.

------
pen2l
Here's a challenge question to folks reading this and learned with the tools
of the trade (my apologies in advance for somewhat hijacking the thread):
consider this video of an endoscopy:
[https://www.youtube.com/watch?v=DUVDKoKSEkU](https://www.youtube.com/watch?v=DUVDKoKSEkU)
\-- say, from 3:00 to 5:00. And I have a bunch of movies (i.e., a series of
images!) and I want to do a 3d reconstruction of this.

It seems super, super difficult... there are free-flowing liquids, and since
this is an esophagus/upper lining of the stomach which is changing in form
quite drastically so often. How would you guys approach this problem?

~~~
ghj
Even more hijacking, I remember thinking medical applications were going to be
the killer apps for VR. I was blown away by these demos almost half a decade
ago [https://youtu.be/MWGBRsV9omw?t=251](https://youtu.be/MWGBRsV9omw?t=251)

Did they ever make it into real life practice?

~~~
Cactus2018
Thanks for linking to Doc Ok's youtube channel!

[https://www.youtube.com/c/okreylos/videos](https://www.youtube.com/c/okreylos/videos)

5 years ago he was active in the Vive VR world

[http://doc-ok.org/](http://doc-ok.org/)

------
tibbon
I wonder how long it's going to be before we're able to run a significant
portion of Youtube video (tourist videos, etc) through something like this,
and generate a huge 3d mesh of the world. Combined with Street View data,
you'd really have a ton of spaces covered.

~~~
pfranz
I believe random videos are too low of a quality. Like this, most of the stuff
I've seen uses constrained videos.

I have seen random still images used for this kind of thing:
[https://nerf-w.github.io/](https://nerf-w.github.io/)

I haven't heard of any equivalent of EXIF for video. That goes a long way when
trying to make sense of random video both for camera settings as well as GPS
location if you're trying to correlate multiple videos.

~~~
tonyarkles
GoPro has a proprietary format that stores live metadata in the videos if I
recall. Maybe it’s called GPX? About 6 months ago I extracted GPS coordinates
from a video using an open source tool.

------
bl0b
Looks awesome. Given it takes position data along with images, how accurate
must the position data be? Could it handle something like sensor drift in the
position data over time?

------
toomuchtodo
For anyone with domain knowledge, how applicable is Google's NeRF work here in
comparison? Is there any overlap?

[https://nerf-w.github.io/](https://nerf-w.github.io/)

[https://news.ycombinator.com/item?id=24071787](https://news.ycombinator.com/item?id=24071787)

EDIT: @bitl: Tremendous, thanks for the reply. Would be amazing to be able to
build these scenes just by walking around scanning a room with your mobile
phone while it records video for processing the frames into scenes (especially
considering mobile platforms with a depth sensor for enrichment of the
collected data).

~~~
nestorD
By default NeRF does not produce a mesh (but one could use marching cubes as
does Atlas) and it requires training a neural network for each scene whereas
Atlas (as far as I understand it) uses pretrained network to process new
scenes.

NeRF would probably produce a much better final result but the Atlas approach
(no need to train something from scratch) is the only one that can hope to be
run in real time which is vital for some application.

------
nickponline
Is there anything that would prevent this approach working on 360 video?

~~~
exit
i imagine a lot of unfortunate artefacts come out of stitching together the
camera views that form a 360 or "spherical" image.

~~~
TaylorAlexander
Well on a camera with dual fish eye lenses for 360 vision there’s some
blurring at the edge where the images are merged together. But actually each
camera separately just has normal fish eye effects, and if both images are
used without blending them together you’d have minimal artifacts. Biggest
issue is low visual acuity imo.

------
jjbolaygaj
Ladies and gentlemen you are looking at the pinnacle of mankind's
technological achievements. The proof?

We can now make tiny virtual cars do stunts off object in the real world:
[https://www.youtube.com/watch?v=9NOPcOGV6nU&feature=youtu.be](https://www.youtube.com/watch?v=9NOPcOGV6nU&feature=youtu.be)

