
Bokehlicious Selfies - rahulrav
http://rahulrav.com/blog/bokehlicious.html
======
rezistik
This is really cool, but it isn't bokeh, it's just blur.

~~~
chongli
Yeah. I don’t see anything here that couldn’t be done in Photoshop with a bit
of masking work.

Here’s an example [1] of how bokeh really looks, using a high quality fast
lens on a decent camera. We’ve got a long way to go.

[1]
[https://upload.wikimedia.org/wikipedia/commons/8/8a/Josefina...](https://upload.wikimedia.org/wikipedia/commons/8/8a/Josefina_with_Bokeh.jpg)

~~~
IgorPartola
So it’s a “find bright spots, bloom them into circles or hexagons or some
such, then make 2-4 offset less opaque circle clones and overlay them offset
by position”?

~~~
chongli
There’s a bit more to it than that. Look near the edges of the frame vs the
middle. Also note that the effect is not binary. It’s dependent on depth as
well as angle. One of the reasons the example from the original article looks
so unrealistic is that the flowers are not at a great depth so they really
shouldn’t be blurry unless the subject is also out of focus.

Additionally: finding true bright spots (vs regular objects that happen to be
white) is difficult with a typical low dynamic range photograph. You’d have a
much easier time with an HDR photo. It still wouldn’t give you any depth or
angle information though.

------
Derbasti
To make the Gaussian blur look better, it needs to be applied to linear
lightness, not to gamma-encoded colors, as it is done here. In reality, the
blur is induced by the lens, on linear lightness, and only gamma-encoded
afterwards.

The effect being that blurred colors mix wrongly, as you can see where the red
from the flower mixes with the green. The transition looks wrong, and not how
a lens would render it.

~~~
dylan604
Shallow depth of field blur is not a Gaussian blur though. Attempts at faking
shallow depth of field with that filter will not result with true bokeh.

~~~
mdciotti
True, but Gaussian or not, convolutional filtering should be performed in
linear light space!

------
rahulrav
This is a quick proof of concept of an idea. This won't work for every use-
case but it worked surprisingly well for a use-case I had in mind. Happy to
answer any questions.

~~~
dillonmckay
So, is the ML part just the segmentation, correct?

~~~
rahulrav
Yes. That is correct.

------
mark-r
I just discovered that Zoom teleconferencing software has the ability to
detect the background and replace it with an arbitrary video in real time. I
have no idea how they do it, but it's as impressive as heck.

~~~
ycombobreaker
Having just used it, it feels like a few factors:

\- a background scene is usually static. if a pixel matches its historical
average (i e. not changing) then it's probably background.

\- if a pixel color matches its neighbors, it's probably the same as them. i
noticed my black shirt tripped it up on occasion, but was all-or-nothing.

\- people are blobs, not diffuse, so try to segment large regions.

~~~
riedel
Yes, it did not work for me sitting with a laptop on a swing chair. It
actually works better with clear static features in the background, but it
seams to be a bit more (cnn?) because it seams to isolate paintings on the
wall for me as well. Wish there was more open source easy to use camera stream
manipulation. Skype also has background blurring.

~~~
MivLives
My work is using Meets, and I second this desire to edit my camera stream. I
didn't find anything even cheap to do it.

~~~
thaumasiotes
A green screen should make it pretty straightforward?

~~~
mark-r
Yes, that would make it trivial. But Zoom gives you the option of using a
green screen or not.

------
joshvm
It would be nice if there was some more information about where the mask comes
from. We want a segmentation map for people, so this technique basically takes
the layer activation map for the "person" class - which is id 0 in COCO - and
you can threshold that to get foreground/background. If you changed the mask
index, this would respond to other object types (so as written this code will
only work for humans).

\--

By the way, Google does some absolutely nuts stuff with this on the Pixel 3
and 4 - they actually calculate a stereo depth map using the two autofocus
sites in _individual pixels_. Essentially some modern CMOS sensors use a
technology called dual pixel autofocus (DPAF), by measuring the response from
two photodiodes in the same pixel, you adjust focus until each pixel has the
same intensity (more or less). If the camera is out of focus, the two
photodiodes will have different intensities.

However what this gives you is two separate images with an extremely small
(but detectable) parallax which can be used to give coarse 3D reconstruction,
and you can segment foreground and background. It's nice because you get a
strong physical prior, rather than having to worry about using a convnet to
identify fore/background regions. (They of course apply a convnet anyway to
refine the result).

[https://ai.googleblog.com/2018/11/learning-to-predict-
depth-...](https://ai.googleblog.com/2018/11/learning-to-predict-depth-on-
pixel-3.html)

[https://ai.googleblog.com/2019/12/improvements-to-
portrait-m...](https://ai.googleblog.com/2019/12/improvements-to-portrait-
mode-on-google.html)

~~~
rahulrav
It’s a simple Gaussian kernel multiplied with a triangular mask.

You can do better by playing with intensities of pixel values as suggested in
the article I linked.

~~~
joshvm
I meant about where this line comes from:

    
    
        mask = masks[0][0]  
    

Presumably 0 is the class ID? For someone new to ML or object detection, it
might not be obvious why you take the first channel here.

Also recent related reading: [https://bartwronski.com/2020/03/15/using-jax-
numpy-and-optim...](https://bartwronski.com/2020/03/15/using-jax-numpy-and-
optimization-techniques-to-improve-separable-image-filters/)

HN Discussion:
[https://news.ycombinator.com/item?id=22590360&ref=hvper.com&...](https://news.ycombinator.com/item?id=22590360&ref=hvper.com&utm_source=hvper.com&utm_medium=website)

------
mdciotti
Awesome! Computational photography is fascinating. A while back, I wrote some
OpenGL shaders to do the same thing, and got stuck when I wanted variable
bokeh based on scene depth. This is easy when you have the depth map (say, in
a renderer). I recall reading about how Apple achieved this using stereo
disparity mapping, which is great if you have the hardware. I hadn't
considered a neural net approach like yours for auto-generating a depth map,
although I now that I look into it, Facebook recently published research on
just that!

[https://ai.facebook.com/blog/-powered-by-ai-turning-
any-2d-p...](https://ai.facebook.com/blog/-powered-by-ai-turning-any-2d-photo-
into-3d-using-convolutional-neural-nets/)

------
amelius
One problem with bokeh is that it disappears when you scale the image down.
Perhaps this technique can help solve that (but I suppose there would be
better ways to do this).

Anyway, the entire technique is also very easy to do manually with Photoshop.

~~~
dreamer7
I don't have much experience with Photoshop. Would it be possible to record
some sort of macro in it to automatically transform any image?

If not, this approach seems to have merit

~~~
inakarmacoma
Yes, Photoshop supports pretty customizable recorded and custom actions. Can
also process multiple files in bulk/batch.

------
dheera
One step further would be to incorporate the blurring, masking, and combining
operations into the network itself, so you can leverage the GPU to do the
whole computation in one fell swoop.

------
kingkawn
Ain’t no lens that would blur the front brim of her hat and her face is clear
like that. Disregarding the nature of the optics you are emulating ruins the
value of the effect

------
a012
OT: I can't understand why this website _want_ to looks like a mobile web
__and__ hides scrolling bar. I'm on desktop and it irritates me a bit.

~~~
RickS
Might be that it's a mobile first theme that's more like mobile only – styling
for the phone and not making any changes for desktop.

~~~
macintux
What’s odd is that I can’t interact with it at all on my iPad. I scroll down,
hit a media object of some kind, and that’s the end of it. Blank space, no
scroll indication, can’t seem to scroll back up to the top.

------
Causality1
I think I'm missing the part of my brain that lets me understand why bokeh
selfies are even a thing. If you're going to remove your surroundings from the
picture, why even bother taking a new picture?

------
smikhanov
His original photo looks way nicer compared to the processed one.

------
kyberias
Page is broken with no scrollbar.

------
Jack000
if we're using deep learning anyways, might as well use a u-net (ie. pix2pix)
to add the bokeh directly - this would require pair training data but would
probably look a lot better visually.

------
lsh
website has no scrollbar??

~~~
dewey
Only works if you scroll while hovering over the "content" part of the website
in the center.

------
marta_morena_23
Wow this looks ugly :D. But still nice work. It's just that the girl is
totally over-processed and the "bokeh", well I hate to break it to you, but
this isn't bokeh, its called "blur".

This is bokeh:
[https://cdn.mos.cms.futurecdn.net/JgrDuxQPCAvgd5VzqiKN5a-650...](https://cdn.mos.cms.futurecdn.net/JgrDuxQPCAvgd5VzqiKN5a-650-80.jpg.webp)

And bokeh is 3 dimensional and surrounds the focus area, increasing with
distance:
[https://www.flickr.com/photos/scottcartwrightphotography/145...](https://www.flickr.com/photos/scottcartwrightphotography/14511003253/)

I suggest to brush up a bit on photo skills, although I agree that the
instragram and iPhone culture can shift your perception of "bokehlicious"
quite a bit.

I don't want to make you feel bad, so I also tell you that Apple's "bokeh" in
portrait mode looks disgusting, doesn't make any sense and breaks the image in
a lot of trivial cases... and they spent a lot more money & effort on it than
you :).

~~~
rahulrav
I agree. My goal here was to get something that look like bokeh if you
squinted at it.

I just wanted to see how far I could take this idea.

~~~
macintux
FYI, the web page seems entirely unusable on my iPad Pro. It goes blank at the
first example, and without a scroll indicator I have no idea what’s going on.

I was able to scroll to the bottom footer, but I can’t scroll back to the top,
so there’s literally nothing but a blank page now except for the footer.

~~~
rahulrav
Thanks for the feedback. Let me fix that.

