
BodyPix: Real-Time Person Segmentation in the Browser with Tensorflow.js - mrburton
https://blog.tensorflow.org/2019/11/updated-bodypix-2.html
======
asymmetric
This seems like it can be used to lower the bar for (repressive) governments
to identify people.

E.g. by making it easier to distinguish parts of the body, identification via
them, or gait analysis, becomes easier.

Are there compelling use cases that make this risk worth it?

Or are we just producing tech because we can?

------
minimaxir
This update is for the 2.0 version.

It's a shame there isn't a Python TF implementation of this (apparently due to
a different file structure between the tf.js model and TF SavedModel?), and it
doesn't sound like it's a priority for the team.

~~~
m0zg
TF 2.0 in general (SavedModel included) has been a giant clusterfuck so far.
As we used to say at Google "the old thing is deprecated, the new thing is not
ready yet" \- internally Google is pretty much entirely like this. I get that
they had to ship it sooner or later, but IMO they shipped 2.0 a bit too soon.
Shipping sooner is fine if you fix the bugs quickly, but that's not really
happening either - some rather potent bugs remain open for half a year or
more.

So at the moment you have a choice of working with a version that's officially
EOL (1.15) or using a version that has several critical features broken or
missing, and on which TF Model Zoo doesn't work. Oh, and EOL version is also
busted beyond belief. Worse yet, Google does not reward maintenance work, and
since this is considered "launched", you can count on the bugs being there for
an extended period of time until Google itself moves to 2.0, which I'd wager
is unlikely to happen anytime soon. Facepalm.

~~~
missosoup
So basically everyone should move to pytorch and ONNX?

~~~
m0zg
ONNX is a turd in is present state as well. Too many unsupported ops and you
can't really run ONNX on e.g. a phone or really anywhere. You could
theoretically convert it into something that does run where you need it to
run, but then you have to deal with the converter's supported ops as well, and
design your net around the intersection of the two sets of ops and their
parameters. If you're doing anything more complicated than basic classifiers,
you're going to run into issues.

That said, I think in a few years with enough elbow grease ONNX could become
pretty exciting.

As to what people should use for research: just use PyTorch. I switched to it
when it first came out, and never looked back, _except_ when I need my stuff
to run on e.g. a phone or something, which is something PyTorch won't help you
with.

------
MayeulC
When I was a lot younger, I used to play softair sometimes with my friends
(wargames with BBs gun). I seriously researched ways to build automatic
turrets.

Rules are simple: shoot at a foe until it raises its hand, that means the
person was shot.

Unfortunately (?), the available software at the time was subpar, and I forgot
about it because of two problems:

* How to identify a foe? It would need facial/body recognition from a few set of examples (team pictures before starting, better if it shots everything else).

* How to know when to stop shooting?

These new technologies would make almost trivial to build such an application,
and make it quite reliable. I guess there is a market for this in softair...
I'm even tempted to have a go at it, BUT... I don't think I will because I now
have to ponder the ethics of building something that can take a gun, identify
targets, aim at them accurately, and fire until they are down. I am usually
all in on open source software, but that's just the kind of things that sound
dangerous to share (same as defense distributed, TBH).

I hope enough persons share the same misgivings, otherwise there is no point
for me to refrain from building this. (Except perhaps spending my time on
project that are actually useful for humankind).

------
lowdose
Since I quit smoking, I noticed how many tv-series and movies are loaded with
cigarettes, alcohol and pharmacy pill consumption. Almost every social event
good and bad on screen is dealt with by drinking a shot of liquor. Is this my
personal paradigm or are we constantly fed with what we should view as our
western culture?

I wonder if it is possible to make an estimation per actor / producer / studio
how many alcoholic, drug, cigarette they consume on average. Would be awesome
to see if there is a correlation with project budget. And maybe even more
important is there an increasing trend of average number of drug consumptions
per hour.

~~~
test1235
comment on the wrong article?

------
the8472
> Person segmentation

Consider the defense applications!

~~~
amelius
[https://en.m.wikipedia.org/wiki/Dazzle_camouflage](https://en.m.wikipedia.org/wiki/Dazzle_camouflage)

~~~
taneq
I think they meant disassembling the actual persons. >.>

~~~
klyrs
I believe gp's point is that Dazzle camo should do pretty well to defeat this
system.

------
mark_l_watson
JavaScript is about my fifth favorite language - way down the scale of
languages that I enjoy using.

That said, I think TensorFlow.js is awesome on so many things: examples are
easy understand and the build system makes them easy to run, many
possibilities for using trained models in browser based apps, on my Linux GPU
deep learning box training models is fast, great documentation, etc.

In some ways I like the TensorFlow.js ecosystem better than the C++/Python
ecosystem.

So far, Swift TensorFlow has been a disappointment for me, but every few
months I check it out again.

------
batmansmk
What can run client-side will run client-side.

------
netwanderer3
Can this potentially be used in 2D animation replacing the manual skeletal
animating processes?

~~~
soylentgraham
That's how we did this;
[https://m.youtube.com/watch?v=7jfZ6zLkWF0](https://m.youtube.com/watch?v=7jfZ6zLkWF0)
(was running live on-set)

~~~
echelon
That's so impressive!

What company do you work for? This is super cool.

Can you talk about the rest of your stack? Or other projects of a similar
nature?

~~~
soylentgraham
[http://newchromantics.com/](http://newchromantics.com/) is my own little
company, (website is criminally out of date) I did the tech,
[http://analogstudio.co.uk/](http://analogstudio.co.uk/) did the art/comp/post
work.

The setup was essentially an app (c++ engine on osx running models), ingesting
camera feeds, outputting skeletons, running sound->mouth poses, to unity which
puppeteered 2D sprites.

On-set we output to a monitor which mixed the real shots with our graphics.

Nothing crazy! But it was a solid setup (except for overheating DSLRs)

------
error265
is there a serverside version of this. I've been looking for somethign similar
in python all day. Actually seems a little buggy seeing this test ->
[https://superanimo.com/bodypix/](https://superanimo.com/bodypix/)

------
m_eiman
Does this work in Safari for anyone? I've tried on iMac, iPhone and iPad.

Only Firefox/MacOS works for me.

------
oh_sigh
Cool demo but how can it figure out that all 3 guys are wearing briefs?

~~~
Rebelgecko
What's weird about the pelvic area coloration is that as far as I can tell it
doesn't line up with any of the body parts they say the segmentation is
looking for ([https://github.com/tensorflow/tfjs-
models/tree/master/body-p...](https://github.com/tensorflow/tfjs-
models/tree/master/body-pix#the-body-parts))

------
mrieck
Wow that's impressive.

Is this library MIT license or something else? I would try using this to
cutout people from images/gifs for my animation platform SuperAnimo:

[https://www.superanimo.com](https://www.superanimo.com)

~~~
minimaxir
Apache (which is sufficiently permissive), like most open-source Google
projects.

~~~
mrieck
Meh - it didn't work that well when I tried it. [1]

Params I used were {architecture: 'ResNet50', outputStride: 32, quantBytes:
2}, with segmentMultiPerson function.

The same images DeepLab handles perfectly. [2] If anyone sees a problem with
the params/method I used let me know.

[1] [https://superanimo.com/bodypix/](https://superanimo.com/bodypix/)

[2] Deeplab:
[https://github.com/tensorflow/models/tree/master/research/de...](https://github.com/tensorflow/models/tree/master/research/deeplab)

