
Facebook Now Lets You Turn Any 2D Photo into a 3D Image Using AI - notlukesky
https://petapixel.com/2020/02/28/facebook-now-lets-you-turn-any-2d-photo-into-a-3d-image-using-ai/
======
folli
Why not directly link [https://ai.facebook.com/blog/-powered-by-ai-turning-
any-2d-p...](https://ai.facebook.com/blog/-powered-by-ai-turning-any-2d-photo-
into-3d-using-convolutional-neural-nets/)

------
Mr_P
Researchers at Adobe published similar work several months ago:

[https://news.ycombinator.com/item?id=20978055](https://news.ycombinator.com/item?id=20978055)

Interestingly, FB folks used a vastly simpler network architecture, but it
probably still works because they have access to so much real data. In
contrast, the Adobe paper had a fairly involved network with pretrained VGG
features, possibly to compensate for only having synthetic data to train on.

~~~
dheera
It's also worth noting that FB doesn't need a lot of accuracy in the depth
data for this to be "good enough" for a simple wobble effect on a phone.

It definitely doesn't need to be held to the same standards that would be held
for a monocular depth estimator for, say, an autonomous vehicle or robot; in
FB's case it doesn't matter whether the tree is 4m or 6m away, and really only
matters that it's roughly behind the dog or cat or whatever.

Separately, depth estimators like this are also useful to fake the depth of
field effects of large-aperture lenses, which I'm pretty sure is used on the
latest Pixel and iPhone in their background-blurring "Portrait" mode; Facebook
could quite possibly include that as an in-app feature to make such effects
accessible to everyone with cheaper phones as well.

------
dTal
>If you’re more interested in how this impacts you, just know this: whether
you want to convert “decades-old family photos” to 3D, or you want to convert
your professional portraits into 3D-like creations, FB now lets you do that.
You just need to upload it using the Facebook app on “an iPhone 7 or higher,
or any recent midrange or better Android device.”

...aaaand that's the motivation for this feature - persuading people to upload
more photos to Facebook.

~~~
xeromal
lol, you say this like it's a revelation. Why would a company build a feature
in their product and not expect people to use their product to use the
feature.

~~~
ThePadawan
There's a significant difference between "this works for all your current
pictures on Facebook" and "you need to choose to upload some specific pictures
on this specific app to use this feature".

The latter will get you a far better information on what people want to see
turned into 3D.

~~~
IAmEveryone
The text is ambiguous, but I would interpret it as "Photos need to be on
Facebook to do this, so those not already on[...]" rather than "This works
only on Photos newly uploaded".

------
sk0g
3D Image ≈ Lovecraftian monstrosity. I've seen a few examples of this, and the
results are mixed in a rather entertaining way!

~~~
raxxorrax
To be fair, raw data from "real" stereoscopic approaches looks fairly similar.
Definitely a good result, but certainly something I have seen before in a
comparable quality. Hard to judge from the examples though.

------
illwrks
Anyone remember an iPhone app canned Seene (I think) that did similar about 7
years ago? It lacked the AI aspect but essentially you took several photos
from different angles and it created some form of depth map with the central
image as a texture (if I remember correctly) I was amazed by this at the time.

~~~
fsloth
Photogrammetry is a commoditized set of algorithms and workflows to do this.

[https://en.wikipedia.org/wiki/Photogrammetry](https://en.wikipedia.org/wiki/Photogrammetry)

There is a good introduction to production use in this GDC presentation from
DICE:

[https://www.gdcvault.com/play/1023272/Photogrammetry-and-
Sta...](https://www.gdcvault.com/play/1023272/Photogrammetry-and-Star-Wars-
Battlefront)

Examples of applications that are usable for individuals include:

[https://www.3dflow.net/3df-zephyr-pro-3d-models-from-
photos/](https://www.3dflow.net/3df-zephyr-pro-3d-models-from-photos/)

[https://www.agisoft.com/](https://www.agisoft.com/)

[https://www.capturingreality.com/](https://www.capturingreality.com/)

And free/open source/academic:

[https://colmap.github.io/](https://colmap.github.io/)

[http://ccwu.me/vsfm/](http://ccwu.me/vsfm/)

[https://alicevision.org/](https://alicevision.org/)

~~~
zimpenfish
[https://get.display.land](https://get.display.land) is also pretty good. It
did a passable job of our local statue[1] in about 3 minutes of capturing +
[some time of their cloud processing].

(Now it's not in the same ballpark as the capture I made with 5 minutes of
taking ~90 photos and the ~20 hours of Meshroom processing but that's
understandable.)

[1]
[https://www.display.land/s/4QeZc8gT77beNLx44OdvDT](https://www.display.land/s/4QeZc8gT77beNLx44OdvDT)

------
TeMPOraL
I remember playing with effects like this ~8 years ago. My first instinct
would be to laugh at the "AI" part, but I'm guessing the point of the exercise
isn't the parallax effect - the point is that they've actually managed to get
a _neural network_ to do that.

~~~
IAmEveryone
I think the point is that it's far better than anything you may have been
doing 8 years ago?

------
mcphage
The picture of Trevi Fountain has a very complicated 3D shape, almost none of
which was inferred correctly.

~~~
sk0g
The article was singing its praises, meanwhile I was trying to figure out
which parts the system _did_ get right...

It's entertaining at least :)

------
m3kw9
I would call that 2.1D the .1 adds parallelax effects

~~~
dheera
It's 3D depth estimation from a 2D image. There is no such thing as 2.1D, 2.5D
unless you are either a salesperson or a theroetical mathematician.

Parallax effects are not 1/10 of a dimension in any sense of the word.

~~~
drdeca
Obviously it does not literally add 1/10 of a dimension. However, saying "2.1
dimensional" gives people the idea of "oh, it is still mostly the same as a 2
dimensional thing, but is a little bit closer (in some sense) to being 3
dimensional.".

------
tehsauce
"convoluted neural network" :)

~~~
0-_-0
As someone who works with them, they certainly seem like that sometimes :)

~~~
bitL
Only in TensorFlow...

------
vidanay
Where "3D image" = amorphous blob.

------
chrisco255
I really want background replacement to be totally seamless. We've got to be
close to that with this technology.

------
spectramax
How many people like watching 3D movies at home or at a movie theater?
Sometimes old fashioned 2D images are just fine. The content, story and
artistic aspects of the image is what matters, not some cool half-baked 3d
trick that is... frankly a distraction.

I don't mean to criticize the research and engineering that went into doing
something like this - it truly remarkable. I refer to the application
possibilities that we might see in the future - imagine the web full of these
3d images, I think that would be a pretty noisy and distracting place.

~~~
netsharc
If you extrapolate where social media apps are taking us, first it was to
"keep in touch" (however superficial) with people, even when it's a tweet that
"they're pooping" or an IG of what they're having for lunch. And then came
Periscope livestreams and Snapchat stories (stolen by Instagram and Facebook),
so you can see a live video of what your friends are doing. FB bought Oculus,
I wonder if their fantasy of the future is for realtime immersive sharing of
experiences, your friend is on top of the Eiffel tower, you put on your
goggles, and boom, you're there with them with a 360 degree video.

At least that's the technological side, personally I think this simulacra of
friendship/socializing is terrible.

~~~
bryan_w
You should watch some of sucks F8 keynotes. He lays out basically what you're
saying

------
fasicle
Would this work for converting 2D panorama photos to 3D for VR purposes,
essentially creating a fake stereoscopic photo?

------
Hoasi
Hey! Here is a novel way to get people to submit their photos to Facebook.

------
saagarjha
I wonder if this could be used to improve background removal tools.

------
simon_000666
Can anyone recommend a good open source version of this?

~~~
mih
Never tried it myself, but this might be an alternative -
[https://github.com/sniklaus/3d-ken-burns](https://github.com/sniklaus/3d-ken-
burns)

------
etiam
I don't need Facebook's permission to do that.

------
reportgunner
Why did we need this ?

Are we now going to remake all pictures into """3D""" and leave the boring 2D
pictures in dust ?

------
gpu_explorer
They can make this very cool yes, but still isn't enough for me to enter
Facebook. They should have more money for privacy.

