Needed to blur the clutter behind me for meetings. Zoom and Google Meets don't support that on Linux. Couldn't find any existing solution that would give me a decent frame-rate for HD resolution without lagging. Wrote this instead.
Relies heavily on GStreamer under the hood for the video processing bits. And mediapipe for the selfie segmentation bit. Obviously need v4l2loopback to emulate a webcam. It's functional, but feedback and contributions are welcome :)
That's good news, but I'm kinda stuck using the web interface because the desktop client doesn't allow sharing single Wayland windows in presentation mode. Unless they've changed that too recently?
> The blurred background option is only available for the Windows and macOS desktop clients, as well as the Android and iOS mobile apps. Desktop clients must meet the "Image only without a physical green screen" requirements.
Nope, no special API what so ever. In fact there's even a Firefox add-on that makes it work[0], Google Meet just doesn't target more than Chromium with this feature.
If you try using it, they'll redirect you here[1], and tell you to:
> Check if your browser supports WebGL 2.0 at webglreport.com and verify that “Major Performance Caveat” is marked as “No”.
...but clicking on that site from Firefox shows WebGL 2.0 support and "Major Performance Cavet" is indeed marked as "no". That being said, simply changing the user agent to Chrome also doesn't work.
So in conclusion, I end up using Chromium a few hours per week purely for Google Meet's blurry backgrounds, and I use Firefox for everything else.
Played with the extension, it does not seem to enable this feature: the option appears, but then there's an error message. Although the other features are fun.
It works for me on Chrome but I had to use Wayland (not Xorg) and then force enable GPU rendering because it's currently blacklisted for Wayland. Possibly relatedly to that blacklist every couple of days (even with rebotos) new tabs in an existing browser window stop rendering content and I have to quit and reload. That doesn't seem to happen on Xorg/without GPU.
Kinda. Google uses the WebAssembly SIMD extension for their segmentation. Not sure if they have a fallback on Firefox, but if they do, it most likely won't perform the same way.
One thing to note about OBS, as much as I love it and use it all the time: if you're using it to record/stream just one source, you're likely wasting a lot of resources. I see many people using it to record/v4loopback their screen or camera, which OBS is not particularly efficient at. Not something you're likely to notice on a modern workstation, but on an older machine or laptop the overhead is definitely noticeable.
Dedicated recording apps for cameras and screens will give you better performance and occasionally a marginally better image since there are fewer steps in the pipeline.
For v4loopback, a short gstreamer pipeline (for those willing to use the CLI) will perform better and give you noticably lower latency - even if some basic cropping is required, like if you're using a capture card with a camera that you can't hide the UI on.
The last time I tried, v4l2loopback wasn't a part of my kernel, at which point I decided it wasn't worth it to tinker with as I didn't know the stability of the mod.
Can't get it to work at the moment - but hopefully with a bit more experimenting, this will replace my current options.
Hadn't realised until reading this submission that Zoom has added blur support on Linux recently - on trying it out it does seem quite poor though - a very sharp edge and quite slow to update.
I believe the Raspberry 4B can act as a USB client device (e.g. a webcam using the Pi camera as video source); could this run on one of those? That would give you an external webcam with built-in background removal, independent of your local setup and video call app and OS.
Right now SelfieSegmentation is just a thin wrapper around the selfie segmentation solution provided in https://google.github.io/mediapipe. It operates on RGB frames, so that's why I need the conversion. The model inference is also done on the CPU. Interestingly, there is a GPU mediapipe graph available, but I haven't looked into what's needed to use that yet.
And yes, Boxfilter is just a wrapper around opencv's boxfilter. This is probably the lowest hanging fruit that could be moved to use GPU.
I want to prank my students. During class, I want to play a video as my video-background, in which a pre-recorded version of me is doing things behind me.
There are many others similar to the above. They all basically work by using the v4l2loopback driver to create a loopback video device. Then you have the program consume video from the real webcam video device, apply a deep learning model that subtracts the background (leaving your face and torso) and replaces it with an image or video of your choice, and outputs the video to the loopback device. Then you configure your video conferencing software (zoom, teams, webex, whatever) to use the loopback device instead of the real one.
It's "easily" done with OBS and a green screen, however having sufficient quality for the prank to work is trickier than it sounds (matching the lighting, getting the good keying quality, avoiding the green cast, calibrating and matching the color etc).
I am not looking for perfection. Just want to get a laugh out of them.
I also just realized that if I place my laptop camera in the exact same spot, while recording the background video and during class, all I need to do is sit on the left of the camera pane during class, and only stay on the right for the background video, then I just need to stitch the two videos side-by-side in OBS and it should work well enough.
I think I achieved my purpose here. Ask a question from a bunch of smart people. Either they give a great answer, or their response forces you to rethink and figure out a simple solution. Thank you HN for being my AI enabled rubber duck.
Yeah, if you don't have a "real" green screen, you can probably shoot a video from the same angle where you come in from one side and stay on that side. Then use OBS's crop filter and keep your live video on the other side.
I just said to hell with it and bought a bright green sheet early on last year since I was spending so much time in meetings and assisting professors with running their classes. That sheet and a couple of cheap clamp lights (plus a decent tabletop mic) made a huge difference in the quality and entertainment value of my telepresence work.
well the easiest way I can think of is just using the OBS virtual camera thing (Unsure if they released it officially on Linux, but there is a plugin for that [1]).
Use chroma key to remove the background and add a video there, possibly on a loop.
Cool, thanks, I look forward to trying this out. Been annoyed ever since my company moved to teams and I can't do all the cool background effects that my windows and mac coworkers have access to.
Is there anything like this for Windows or Mac? Many third party videoconferencing solution don't offer blur if you're interviewing with an employer, using Jitsi Meet, etc.
Along the same lines, an issue I was interested in a while back, has anyone written a script in Linux to get the webcam to dynamically crop to faces/people?
Relies heavily on GStreamer under the hood for the video processing bits. And mediapipe for the selfie segmentation bit. Obviously need v4l2loopback to emulate a webcam. It's functional, but feedback and contributions are welcome :)