
Background Matting: The World is Your Green Screen - hr7161
https://grail.cs.washington.edu/projects/background-matting/
======
laumars
This does look like an interesting step forward however I’ve found the biggest
limitations of previous techniques to be

\- people who are using occupying a wider z-axis (for example leaning forwards
in the camera or who have arms in front of them)

\- people holding objects like cups

How well do your method handle those kind of situations?

~~~
illumanaughty
I'm not sure how either of those situations would trip up the system they're
using. For a system trained on the background image, what difference does it
make if the subject is holding a cup? The cup is not the background image, and
it would be obvious in the same way that it's obvious the subject isn't the
background image.

~~~
gnramires
He might be referring to transparent/translucent/refractive objects, like a
glass cup. Supposedly this technique can manage the transparency, but not
refraction (and maybe the refraction could trip the transparency into
failing).

------
BubRoss
This is broadly called natural image matting if anyone is curious enough to
look into the last 20 years of research.

~~~
vanderZwan
Thank you!

(wouldn't it be nice if every time a research topic pops up here that there
would be a small list of the essential keywords to look for more background
information?)

------
shubidubi
Is there an easy way to stream it to Zoom/Slack etc...? Will be nice to use it
as a virtual camera source

------
myridium
Is it fast enough to operate on a live video feed from, say, a webcam?

~~~
freeone3000
Yes. OBS offers it. Teams and Zoom have guesswork-based imitations.

~~~
myridium
And where can I find the OBS implementation of this work?

~~~
kumarm
[https://github.com/obsproject/obs-studio](https://github.com/obsproject/obs-
studio)

~~~
rahimnathwani
I found the chroma key implementation, but this seems to assume you have a
green screen or similar: [https://github.com/obsproject/obs-
studio/blob/master/plugins...](https://github.com/obsproject/obs-
studio/blob/master/plugins/obs-filters/data/chroma_key_filter.effect)

I thought GP was asking for an implementation of background removal without a
green screen.

~~~
myridium
Indeed. I'm fairly sure it's not in OBS. Which is why I asked for a link to
where it was implemented. Surprisingly I didn't receive one...

------
based2
And Apple PhotoBooth deprecated this functionality too 32 bit debty.

~~~
asutekku
Looking at the demo released, this seems to work much better than apple’s
implementation.

------
pgt
Very cool. Next step would be to emulate lighting in the target scene, but
that probably requires pose detection and facial landmarks for accurate
shading.

~~~
Cerium
I have been thinking that a killer zoom product would be a USB controller for
an rgb led strip that helps match color with your virtual background.

~~~
Wistar
Very good idea. And change in real-time if the background is dynamic. And
allow the user to set styles such as warming the FG subject, shimmering as if
there is a fire or candlelight in the room, etc.

------
ollifi
Anybody know of any work done to improve greenscreen keying? The current old
school techniques work quite poorly and require so much manual work. I would
imagine with the new work coming out with neural nets etc. there would be
possibilities for improvement. This is very cool work and good for certain
applications, but seems to produce similar problems like greenscreen to some
edges.

~~~
dylan604
Modern green screen plugins/filters are miles better than what they used to be
to the point that if the keying is hard, then the image must not have been
produced well. By that, I mean an evenly list background (no light fall off
producing gradients). Proper lighting of the subject. Proper distance from the
background (helps reduce the edges and color tinting).

~~~
ollifi
I work quite a bit with green screen keying. I see the same keylights,
ultimattes and primattes used still even in big productions I have worked.
Fixing the key can take weeks. Maybe industry is bit conservative and I
haven't seen cool new stuff bubbling under, but would love to have new tools
in the toolset to approach difficult shots.

------
kxh6
I recently tried working on this with OpenCV but it didn't quite work like I
expected. I had problems with artifacts caused by my own shadows

------
amelius
Does this work with reflective objects in the foreground, such as a car?

------
web007
What's novel about this?

If you have a background picture, you have all the info you need to identify
your subject - just plain subtraction. I think this is what the Photo Booth
app on my circa-2012 MacBook does, quite effectively.

~~~
KaiserPro
Ex-VFX person here.

Green screens are crap with hair, because its translucent the green/blue
bleeds through which means that it has to be cleaned up by hand.

Then there are the situations where there isn't a green screen. Again manual
cleanup is required. Each frame needs to be cut out by hand. 24 times a
second.

The same with a difference matte. Cameras are noisy, so there is constant
noise in the alpha channel. This makes the effect look wobbly and cheap.

What this method does is pull a key from a difference matte, and makes it look
good.

~~~
jononor
Are there no tools that automate the cleanup of green-screen artifacts?

~~~
jiofih
The green screen software itself is that tool. The parent is saying it has
limitations

------
aledthemathguy
That's @!#!@$ magical!

------
dang
The originally submitted URL
([https://www.catalyzex.com/paper/arxiv:2004.00626](https://www.catalyzex.com/paper/arxiv:2004.00626))
points to

[https://github.com/senguptaumd/Background-
Matting](https://github.com/senguptaumd/Background-Matting), which points to

[https://grail.cs.washington.edu/projects/background-
matting](https://grail.cs.washington.edu/projects/background-matting), which
points to

[https://arxiv.org/abs/2004.00626](https://arxiv.org/abs/2004.00626), which
points to

[https://arxiv.org/pdf/2004.00626.pdf](https://arxiv.org/pdf/2004.00626.pdf),
which is inlined at the originally submitted URL. I'm not sure what's going on
here, but on HN the convention is probably to link to the project home page
first, and after that maybe the Github page and if neither of those exist, to
the arxiv.org homepage (but not the pdf since those change with each
revision). So I've changed to the project home page for now.

~~~
myridium
Are you suggesting it is an advertisement for catalyzex.com?

~~~
dang
I wasn't suggesting anything, but having just looked at the submission history
it seems clear that it's promotional. The HN community doesn't favor that.
It's fine to submit your own site or work occasionally, but not to use HN
primarily for promotion.

Also, the submitted title ('Zoom’s virtual background swap but better. DL+GANs
for background replacement') was too promotey.

~~~
myridium
I agree with you.

FYI your first sentence effectively says > I wasn't suggesting anything, but I
was suggesting exactly that.

------
f_allwein
Peripheral: What is the benefit of halving these artificial backgrounds? Apart
from „it’s fun“, which wears off after about a minute? In my experience (Zoom
Meetings), there’s blurring/artefacts around the edge of the head and the
image quality seems to suffer as well.

I had a meeting where one participant uses an actual green screen, and the
difference was remarkable, with none of the issues above.

~~~
onion2k
There are two parts to background matting. The first is removing the existing
background and the second is replacing it with something else. Removing the
background improves the focus on the foreground - people watching can see you
better and they'll listen more closely because they're not distracted by
what's behind you. The second part, replacing the background with something
else, might be done because you don't want people to see where you are, or
because you want to overlay your foreground video on a presentation. Being
able to pretend you're on a holodeck or a desert island is a trivial use of
the tech.

