
Nvidia Broadcast App - ryneandal
https://www.nvidia.com/en-us/geforce/news/nvidia-broadcast-app/
======
tsumnia
Overall I'm very impressed and plan to incorporate the Background Blur into my
lecture videos (since I had to create a 'decorative' background for them
anyway).

So everyone can see how it looks with a makeshift lighting setup (kitchen
lights, overhead lights, and a 3 bulb lamp).

Background Removal:
[https://i.imgur.com/e3y4kwX.png](https://i.imgur.com/e3y4kwX.png)

Background Replacement:
[https://i.imgur.com/AajgJIf.png](https://i.imgur.com/AajgJIf.png)

Background Blur - Low Setting:
[https://i.imgur.com/CeXmneX.png](https://i.imgur.com/CeXmneX.png)

Background Blur - Max Setting:
[https://i.imgur.com/OoJ76ow.png](https://i.imgur.com/OoJ76ow.png)

Frame Tracking -
[https://i.imgur.com/wQ8IOq1.mp4](https://i.imgur.com/wQ8IOq1.mp4)

Full Album: [https://imgur.com/a/RWECpd4](https://imgur.com/a/RWECpd4)

EDIT: As a followup to a question my friend asked - memory wise it is sitting
idly at 105MB; running OBS it went to 670MB.

EDIT2: The blur effect holds even if you move to the background as well
without sharpening the objects in the background. It did look a little "off"
but that's mostly because a sharply rendered body interacting with a blurred
environment looks off.

~~~
iseanstevens
Nice smile :)

~~~
tsumnia
Gotta look good for all those movie producers that just found their next big
star :D

------
resoluteteeth
> For those of you that want to try out the AI noise removal capabilities but
> aren’t ready to upgrade to an RTX GPU yet, we have also patched RTX Voice
> with support for NVIDIA GeForce GTX GPUs. Though, of course, your mileage
> may vary on older cards.

That's a strange way to say "yeah we admit that we were just blocking older
GPUs in the installer and you guys noticed so decided to stop."

~~~
motoboi
Well it kinds looks like your car manufacturer stating in the manual that your
car cannot tow.

You know it just a matter of installing one, but they basically don't want to
spend time supporting something they have no interest in.

~~~
bluGill
In the case of cars often a car can tow, but it shouldn't. I know someone who
burned out a transmission because the trailer was overweight (in this case it
was a truck that the manual said max tow weight was 1/4th what he was towing).
I haven't seen it myself, but my EMT friends have had to respond to cases
where someone was towing

I work for John Deere, it is well known that ECU code can change the horse
power of our tractors. What isn't as well known is the tractors with higher
horsepower ECUs have more warranty work, and that is a case where we engineer
the system for the change (there is more than an ECU changed in getting higher
horsepower if you buy it that way from us because we don't want warranty
claims)

If you know what the real limits of your system are you will sometimes
discover the stated limits are less than the real limits. However other times
the stated limits are there for a reason.

~~~
andybak
Give people the facts. Let them decide but make sure you're protected from
supporting idiots.

Better than coming out of it looking like the evil dudes.

EDIT being most well known outside your sector for suing your customers is
generally a bad look

~~~
G4E
It sadly doesn't work like that. Idiots are idiots. "You sold me this truck,
and it doesn't work anymore.

\- Well, we stated that you shouldn't do what you did...

\- It doeesnnn't work !!!"

Ensue time, energy and money lost to defend yourself against idiots who don't
listen.

Edit : I don't know anything about trucks or John Deere. But you can see this
type of "artificial" limitation with simple unlock pretty often. For example
the cpu overclocking community : the manufacturer doesn't officialy support
overclocking, but they give hints and some advice if you want to do it anyway.
You fry your cpu, you're on your own.

------
syntaxing
You should see what Nvidia does for CAD software...They used to force you to
use Quadro cards only (which is maybe 2X-3X price of their consumer version)
even though the GT/X series worked just as good. People figured out how to
change the registry setting to make it work until they finally caved and
supported it "somewhat" (where if it breaks it breaks and this was maybe 5
years after the registry hacking). Nvidia has always been notorious for making
people use the "good stuff" (expensive) even though it's not really required.

~~~
TomVDB
> Nvidia has always been notorious for making people use the "good stuff"
> (expensive) even though it's not really required.

Do you think that these Quadro features materialized out of thin air, or did
they require engineering effort and additional silicon area?

If the latter, is it reasonable for Nvidia to expect to be compensated for
that additional effort (which is born by, say, 2 orders of magnitude lower
volume) ?

Additionally, would you be happier if Nvidia simply removed those features so
that no home users would complain anymore (at the expense of Quadro users who
would lose the ability to use real features.)

And, using my standard to-go-to argument, are you similarly upset that
software companies are charging different amounts of money for different tiers
of a software product (e.g. Microsoft Office Home vs Professional), even
though these different tiers are only enabled by a license key? If not, why?

~~~
syntaxing
I personally think there's a pretty big difference between being compensated
for their work and using software to limit where the hardware can be used (and
Im not saying Nvidia is the only company that does this, like Tektronix who is
even worse for it). This differs than a purely software product like you
mentioned between home vs professional usage. When I buy a GPU, I expect to be
able to use it as a....GPU regardless of the software it connects to. Sure, I
get it, there is some additional software work such as CUDA and driver stuff
depending on the package but at a basis with something like DirectX, this
should be more or less software agnostic. Unfortunately our market is heading
towards that direction (e.g Tesla and John Deere) where even "unlocking" the
features via hardware means is a violation of DMCA.

~~~
babypuncher
It's not that simple though. It's cheaper to engineer one chip that does
everything then selectively disable certain features for lower end SKUs.

Going to the effort of making different hardware for the lower end SKUs that
completely lacks these features would end up raising prices for consumers
because it makes engineering and manufacturing more costly.

The other alternative is they just don't offer lower price SKUs and make
everyone buy top-end parts.

------
ryneandal
Due to having to keep an eye on my kids during virtual school, I'm stuck in
the family room which is incredibly noisy with toys and furniture. I've played
with the software for about 5 minutes now and think the virtual background
functionality works incredibly well:
[https://imgur.com/a/0ByEYF9](https://imgur.com/a/0ByEYF9)

~~~
codezero
Especially given that you have a bright background light, also that it cut
through the space between your headphones. Pretty neat, and better than zoom’s
silhouette guess.

~~~
ryneandal
Yeah, no issues with the shades drawn on the window, it's impressive.

I hadn't noticed the space between my head and headphones prior to this
comment, great catch. I'll definitely be using it during work meetings now,
I'm sold.

------
zaroth
The Broadcast app offers three AI-powered features:

Noise Removal: remove background noise from your microphone feed – ...

Virtual Background: remove the background of your webcam feed and replace it
with game footage, a replacement image, or even a subtle blur.

Auto Frame: zooms in on you and uses AI to track your head movements, keeping
you at the center of the action even as you shift from side to side...

These features can be used beyond game broadcasting as well — from video
conferencing at home with Zoom, to gaming with friends on Discord.

------
movedx
I did a quick demo here:
[https://www.youtube.com/watch?v=aoa0aB1AXtI](https://www.youtube.com/watch?v=aoa0aB1AXtI)

Hardware: Sony A6000 Mirrorless camera; 2x Elgato Key Lights for illumination;
GameCapture HD60 S; RTX 2080 Super; OBS for the video feed and recording.

~~~
wiradikusuma
I have Sony A6000 with Elgato too. Do you mind showing your setup esp. camera
placement? I'm curious your eyes can look at the audience while in front of
PC.

~~~
Godel_unicode
By looking at the camera while recording, watch the first few seconds again.

------
sinak
For anyone not using an nVidia graphics card, I highly recommend
[https://krisp.ai/](https://krisp.ai/) for the voice cleanup part. Works
wonders.

~~~
nerdbaggy
Do you know of any other companies offering a product like krisp? They seem to
be one of the only players.

~~~
emcq
BabbleLabs is the closest but they got acquired by Cisco.

Google Meets also has similar tech
[https://support.google.com/meet/answer/9919960?hl=en](https://support.google.com/meet/answer/9919960?hl=en)

Audatic was really cool but they got acquired by a hearing aid company.

Whisper.ai has very similar processing but they are also focused on hearing
aids.

------
tomasreimers
Interesting, I wonder if on a meta-note we're seeing an evolution of Nvidia's
strategy. For a while, it was hardware, then it was leveraging that hardware
to create developer-facing APIs (CUDA being a driver for many ppl in ML to buy
Nvidia in particular), and now it seems they may be trying out leveraging that
compute to create specialized, end-user facing services.

Are there any more examples of this?

I wonder if Nvidia will try to move upmarket to own the best UI powered
services (e.g. if you want a camera with a virtual background, it has to be
nvidia or something similar)

~~~
01100011
I can't seem to find a single page that lists all of Nvidia's current vertical
software stacks, but they are legion. Aerial(5G), Metropolis(smart cities),
Drive(self driving), RAPIDS(big data?), Clara(genomics), IndeX(visualization),
Isaac(robotics)... etc.. etc...

They basically have to move towards vertical software stacks requiring their
HW. GPUs are becoming commoditized, with integrated GPUs from AMD and Intel
becoming good enough to play modern games now. 8k(haha) may keep things
interesting for a while, but eventually AMD/Intel will compete there as well.

I don't know about user-facing apps though. So far, most of what they've
released(i.e. Gaugan) seem more like demos.

~~~
person_of_color
They are becoming the Apple of B2B compute. And Shield is their killer
consumer app. Watch out for FAANNG

------
nl
Just noting that the Turing series of cards and later (which isn't just RTX,
but also the GTX16* the the T4) _does_ have specialised hardware which helps
with the video tracking side of this.

They have hardware optical flow[1] which doesn't seem widely known, but would
absolutely help with parts of this - especially the reframing.

[1] [https://developer.nvidia.com/opticalflow-
sdk](https://developer.nvidia.com/opticalflow-sdk)

------
jeena
Only a windows version, meh.

------
localhost
I'm running build 460 of the NVIDIA drivers on Win 10 because I'm using the
DirectML support for WSL 2. It appears that Broadcast doesn't work with this
as it tells me "cannot start service" each time I try to launch the Broadcast
app.

Has anyone else gotten this to work, and what build of the NVIDIA drivers are
you using on Windows?

~~~
lostmsu
> I'm using the DirectML support for WSL 2

How's that going for you? In my tests (directly on Windows host), tensorflow-
directml was about 6x slower than tensorflow-gpu (e.g. with CUDA/cuDNN).

------
brainless
Here is the video explaining it nicely with demos:
[https://www.youtube.com/watch?v=GRFjfGH87Dk](https://www.youtube.com/watch?v=GRFjfGH87Dk)

------
raobit
I have 2 Q's

1\. As in video on background removal , why was the chair not blurred and
because generally in background removal only the person is visible and all
rest are blurred, how is it working?

2\. How Background Noise removal works, i mean how it is so smoothly removing
background noise.How it is able to distinguish which is the primary and
secondary,i know it's AI but still want to know the logic and basic
understanding.Thanks

------
redisman
How mature is this software? I tried to use RTX voice on an important call and
it was not working at all causing me to be late and left a bad taste for these
for serious use.

~~~
ShamelessC
I've had no issues with it. The real problem was Windows 10's extremely
frustrating and confusing audio engine.

------
cptskippy
I just updated my Quadro RTX 4000 to the latest available driver and it's only
R450, when I try to install Broadcast it says I need R455 or newer.

~~~
babypuncher
Check for beta drivers. If I recall, Nvidia only cuts new "stable" Quadro
drivers very rarely, putting them through very rigorous validation since they
are pro-level hardware.

~~~
cptskippy
The latest driver was released 8/18/2020 and Broadcast's system requirements
say they're supported...

There don't appear to be beta drivers.

------
brian_herman
This seems better than RTX voice I have my ears blown out when someone uses it
on discord and it crashes.

------
suyash
Anything similar like this exists for Mac?

~~~
godelski
Or linux?

~~~
jcastro
Check out
[https://github.com/lawl/NoiseTorch](https://github.com/lawl/NoiseTorch)

------
gavinray
Odds that this winds up with a Linux release eventually?

=(

~~~
Godel_unicode
I'd say the chance is roughly the same as the percentage of Nvidia customers
running desktop Linux, I imagine. The three of you must be used to this by now
though?

~~~
gavinray
My RTX-2060 runs well on Pop_OS!/Ubuntu

I've had pretty much nothing but good experiences running Nvidia on Linux the
past 4-5 years, barring one strange issue with multiple monitors, a
DisplayLink adapter, and trying to rotate a screen sideways.

I can see fiscally how it doesn't make a lot of sense for them -- but why
write a platform-dependent implementation and not use code that's generic?

Is the problem the GUI? Build it with Qt or something

------
Dahoon
Ah yet another Nvidia thing that would work everywhere but is blocked in
software because greed. Remember how raytracing was supposed to only work
properly on RTX but when people got mad suddenly it wasn't only RTX cards? And
now we are even seeing it on the old and slow PS4. Next they'd likely do
something crazy like require unnecessary hardware in screens or something. Oh
wait..

------
Ecco
Noise reduction, background removal. Technologies that have been around for
ages. But now apparently it's "with the power of AI" so it's newsworthy?

~~~
echelon
Machine learning for signal processing is no joke. It's an order of magnitude
improvement. The approach is different, too. Instead of approaching the signal
mathematically with Fourier transforms and such, you approach it as part of a
pipeline where you extract high level components such as pitch and timber. The
ML model learns how to do that.

This field is going to change everything. Look what I've been able to do with
some of the recent ML innovations: [https://vo.codes](https://vo.codes)

~~~
nullsense
The next layer that would put the realism off the charts is ML that's able to
replicate a given speakers idiolect, not just their voice.

Mimicking people's ideolects is a hobby of mine, and I typed in phrases that
Dr Phil would say, and the voice itself is good but the delivery is off enough
that I know it's definitely not him.

Im sure it all get there one day.

