
Real Time Person Removal from Complex Video - aliabd
https://github.com/jasonmayes/Real-Time-Person-Removal
======
dkdk8283
We’re very close to not being able to trust video, I don’t think many people
outside of tech realize how close we are.

Imagine someone adding your face to a video of a crime.

~~~
mhandley
Pretty soon I think there will be a market for CCTV cameras that use a tamper-
proof module to sign the video output, with a unique key and a key chain back
to the manufacturer. Video evidence simply won't be admissible unless it's
signed, and you can present the undamaged camera in court.

~~~
AnIdiotOnTheNet
> Video evidence simply won't be admissible unless it's signed

Courts today readily admit the testimony of eye-witnesses, which ware
notoriously fallible. It is a common misconception among tech people that all
systems operate in rational ways, but this is only really the case for
machines.

~~~
whatisthiseven
> It is a common misconception among tech people that all systems operate in
> rational ways, but this is only really the case for machines.

Very elegant and concise. I would say this extends to all deeply complex
fields: engineering, physicists, quant, etc. They all have a mindset of "I can
understand the whole system because they have logical components that build on
top of one another" and so therefore expect and demand that all other fields
do the same.

~~~
Mangalor
To be fair, wiser engineers seek to discriminate between the rational and
irrational parts of a system.

------
eskaytwo
The corollary to this is that people’s faith in video will be so low, that
actual video of a major event can be dismissed as potentially fake - and the
more extreme the more easy to dismiss. The sword cuts both ways: false
negatives being accepted as true and true positives being dismissed as fakes.

~~~
mlrtime
How is this any different than a picture? We have been making fake pictures,
editing, airbrushing, photoshoping for years.

~~~
tw1010
Yeah I don't buy this idea at all. People freaked out over the ability to edit
photos in dark rooms decades ago. Trust didn't deplete because pictures got
tied to identities which in turn is tied to reputation (or by verifying "hey
is that picture of the president k-wording a person with a gun real" with a
trusted institution or something), etc.

There's other cool stuff that will come of this but the fact that this is the
idea mentioned _every single time_ shows a severe lack of imagination in our
sphere.

~~~
lonelappde
What do you mean trust didn't deplete?

Faked images are super popular and effective and are used to reinforce biases.

Major outlets like Fox News that traffic in false information have bad
reputations and also good reputations among others.

You are right that this is isn't a technical problem though. Why bother faking
a video or photo when fake words (misquotes and lies) are 10000 cheaper and
just as effective?

------
elboru
Just yesterday I watched a video that was difficult to detect it was fake, it
was too funny to be true though. Anyways I couldn't tell just by looking at
the video. The video showed Mexico's president next to a phallic shaped
cactus.

I like to think I'm good detecting fake videos or images, but I guess now I'll
have to check the sources every time.

Fake (possibly NSFW):
[https://www.youtube.com/watch?v=4r0TCQrfMq4](https://www.youtube.com/watch?v=4r0TCQrfMq4)

Real: [https://youtu.be/31KzD5a3KS4?t=88](https://youtu.be/31KzD5a3KS4?t=88)

~~~
rolltiide
That's just a composite though. Nothing currently novel there.

------
Traster
I was kind of disappointed with this. I may be misunderstanding this but
essentially it seems to use machine learning to identify body parts and then
just plasters over the background. Now what I would have expected from a
'complex video' would at the least be a moving camera - in which case you
actually have to build a model of the environment ala GuaGAN and generate from
it.

~~~
dannyw
Yes. It’s best to underpromise and overdeliver. If you’re not really anywhere
near SOTA, then be accurate in your description, eg “Real-time person
inpainting from video in JavaScript”

------
xiphias2
It would be great to see what the more advanced (but bigger and slower) ResNet
model is capable of:

[https://github.com/tensorflow/tfjs-
models/tree/master/body-p...](https://github.com/tensorflow/tfjs-
models/tree/master/body-pix)

------
fudged71
I work in a gym, where lots of people take video of themselves doing exercises
(for instagram, or form checks), despite the gym's policy for privacy reasons
(other members in the background).

There's an immediate need for an app that will remove the background of people
for privacy purposes.

~~~
MiniGod
Combine this with facial recognition to only remove unrecognized people (or
opposite)

------
lowdose
Commercially it would be more interesting to do active background removal. If
every Youtuber has a professional background like a news anchor they would
immediately look legit.

~~~
bsenftner
Background replacement is a commodity motion graphics tool. It has been around
since the 90's, and exponentially better than this work here.

~~~
aantix
But against an arbitrary background?

I’ve seen real-time background removal for video and for still photos, but
never gets it perfect. Too many artifacts.

That’s why the green screen is still utilized.

~~~
bsenftner
Yes, how do you think automated wire and stunt performer rigs are removed?
Think of all the VFX you see; an immense amount of visual effects are
animation basic principals applied to scraps of previous and past frames for
the composition of the current frame.

------
mackrevinack
this would be very useful in busy tourist spots where you want a video of
something, the leaning tower of piza for example, but minus all the annoying
people pretending to push it over

~~~
olivierduval
Isn't it what "long exposure photography" (and photo stacking in photoshop) is
used for ? ;-)

I mean: if there's too much crowd, you have to wait a long time to get every
little part of what's behind. So I'm not sure that ML will be useful except
maybe to detect "humans" and decide what parts need to be replaced

~~~
bangboombang
Yes! Just take a lot of photos over a couple minutes (depending on how busy
your scene is) from a fixed position, or just a video if you're lazy, then use
imagemagick and combine them using "median" (NOT average). It's not always
perfect but can deliver most of the time. That way even a command line dork
like me can do it. :-)

~~~
ansgri
A similar simple trick allows to reduce reflections from photos of flat
objects under museum glasses (books, pictures, coins): take several photos
from slightly different angle, co-register them using some panorama assembly
techique (I've used a simple Python OpenCV script) and merge with minimum or
some bottom quantile, since reflections are additive.

------
bsenftner
Looks like mid 90's background replacement extremely early work. This type of
thing, and exponentially better - as you know from feature film VFX - have
accomplished these types of operations decades ago, and have published how it
is done, and commodity versions of this logic are in open source packages like
blender. Deep Fakes and Style GAN has people all excited, but this type of
effect does not need ML and are fully features digital toys now. Hell, after
40 years as a developer of games, 3D graphics, interactive video, 3D games,
feature film VFX, a digital double creation service, and now FR I can say with
authority: this work is nothing.

~~~
superdisk
It's interesting though because an amateur slapped it together quickly rather
than an entire industry over decades.

~~~
bsenftner
I wrote one myself back in '93 that was used in talking head videos by a
documentary company to place their talking heads into the locations of their
films. The essential logic is quite simple, leveraging the fact that you've
got previous frames.

------
paulsutter
The earliest cameras around 1830 already removed people, perfectly. Exposure
times were as long as 8 hours, so people were invisible. only buildings,
trees, etc appeared.

Link below includes images of streets with people “removed”, and has the
oldest known image of a person, a shoeshine that stayed in one place long
enough to be seen

[https://www.livescience.com/60387-oldest-
photographs.html](https://www.livescience.com/60387-oldest-photographs.html)

~~~
dTal
It's arguable that 8 hour exposures count as "real time", however :)

------
edf13
What’s impressive is that this is done in JavaScript!

~~~
dmd
Why is that impressive? JS is one of the most powerful and performant general-
purpose languages in common use.

~~~
Keyframe
performant?

~~~
dmd
Ugh. Yeah, that's an awful word.

------
theemathas
I have doubts this hasn't already been done given that the background is
static.

~~~
bsenftner
Yes, 35+ years ago. Background replacement is a commodity motion graphics
tool.

------
uyuioi
Bank heists just got that much easier

~~~
TuringTest
Sure, you just need to hack into the camera hardware system and inject a
browser-specific machine learning library. Easy-peasy

------
jackallis
is it just me or arent we supposed to be terrified of this kind of
development? Cringe.

~~~
johnkpaul
You mean specifically because of the real-time-ness of it? As far as I
understand, with professional video editing tools and skills, this camera-in-
fixed-location removal is trivial.

------
greenbean4
Has "Real Time _Background_ Removal" already implemented?

~~~
penagwin
Yeah it's been around for a long time - although many of the earlier version
were rather awful. There's different methods, and not all of them use neural
networks.

But there's software to do it with a decent camera that works okay, you'd be
better off with a green screen though in most any situation.

------
lettergram
I think this can be improved if edge detection was applied. At the moment it
appears to be a bounding box, this lets shoulders and stuff on the edges
appear out of box

------
mfer
Making provenance of software and data easier will be of increasing importance
in the future. We need to be able to prove things have not been tampered with.

------
yozel
This project should have the name "Stalin"

~~~
coribuci
This will be a praise for him. Unfortunately censorship has become common
practice around the world and it did not started with Stalin.

------
twism
Reminds of the movie "Rising Sun"

------
kashishgrover
This is what they used to remove Bojack from Horsin' Around :O

------
cartercole
i think i saw this in a black mirror episode

------
alasdair_
Stalin famously had people that had “left” his administration airbrushed out
of photographs.

I don’t think it will be long before current-day dictators start doing the
same thing with old footage containing previous allies they no longer wish to
be associated with.

~~~
eb0la
Not just people, the famous 'Red Flag over the Reichstag' photograph was
modified (added smoke and removed watches on soldier arms).

