
Pose Animator: SVG animation tool using real-time TensorFlow.js models - hardmaru
https://github.com/yemount/pose-animator/
======
crazygringo
This is like creating a live version of author drawings, e.g. of New Yorker
contributors:

[https://www.newyorker.com/contributors](https://www.newyorker.com/contributors)

How interesting would it be to have your own live emotionally expressive
avatar for videoconferencing, when you don't want to worry about your hair,
makeup, lighting, or general visual state at all?

~~~
aantix
IOS has the animated animal characters that mimic gestures pretty well.

~~~
eggy
I was playing with OpenFrameworks and Kinect years ago, and you could pretty
much do this, so I am curious why you would TF.js instead of simple OpenCV or
other libraries that don't need to to use ML or DL. Or am I mistaken, and it
is using simple bits of TS.js?

We had a large, flip-disc (or dot) wall at the site in Macau in 2012 that we
purchased from a company that was a wall of black and white discs or dots that
would flip to create a cool effect by tracking people in front of the wall
with Kinect units in real time. It also made a cool clicky sound like old
train/airport physical arrival/departure boards did.

I had a feature on my HTC phone or early Skype over 10 years ago where a
cartoon cat mimicked my mouth, eyes, and head movement live on camera, which I
can't find the reference to, but I have a screen recording of it when talking
with my kids in the US from Macau.

I also remember using animata, a software that animated 2D puppet-like cut-
outs to the music, I played with over 7 years or so ago, that was really cool
[1 YouTube].

[1]
[https://www.youtube.com/watch?v=Dz8OMxB8m_M](https://www.youtube.com/watch?v=Dz8OMxB8m_M)

~~~
RobertKerans
Re the feature in HTC or Skype, it was Skype, there were a set of novelty
faces that mimicked the movement (I can only remember the dog, but there were
a few, some more effective than others).

------
rw2
Wow, this would change how animation is done completely! If there existed an
easy way to create animated character cartoons. It would launch a thousand
southpark/rick and morty type of shows. A team with 10k can launch a show.

~~~
nkozyra
I believe Adobe has had this for a long time in character animator

[https://www.adobe.com/products/character-
animator.html](https://www.adobe.com/products/character-animator.html)

~~~
rw2
Good to know, it's a awesome feature I am surprised I don't see more
successful shows made with this on reddit and youtube

~~~
jjeaff
That's because the animation is very much the least important part of a
successful show. It's the writing and voice acting. South Park is a great
example of this. It's successful because it is funny and edgy. The animation
could have gone a lot of different ways and had similar success I think.

~~~
rw2
I think fundraising is the hardest part of a project, especially for
creatives. If a group of writers can wrangle some amateur voice actors to
work, (Which I think would be the lowest paid actors) it should be easy to get
a project up.

------
p4bl0
It would be cool privacy-wise to have some sort of virtual webcam that could
share such an animation rather than what the webcam actually sees.

~~~
sho_hn
This is fairly easy to wire up on Linux using v4l2loopback and pyfakewebcam.

I am currently using a little setup that uses OpenCV to acquire frames from
the real camera, TensorFlow/BodyPix to compute an alpha mask for the
foreground (me) and then OpenCV again to transform and composite myself behind
news desks and into car infotainment screens and the like, eventually writing
it to a virtual webcam I can use from Zoom (over its own virt bg feature this
adds the layering and perspective transforms), Jitsi, Teams, etc.

The above looks like another fun thing to add. Time to go full _Who Framed
Roger Rabbit?_ ...

~~~
sbarre
Do you have any more details written up anywhere on how you do this? That
sounds like a great project.

~~~
pdsouza
There was a recent technical write-up about something similar:
[https://news.ycombinator.com/item?id=22823070](https://news.ycombinator.com/item?id=22823070)

------
TazeTSchnitzel
> This is not an officially supported Google product.

Does that indicate the author is a Google employee who happened to make this
in their free time, and has to say this somewhere by Google policy?

~~~
jpalomaki
"Unless your project is an official Google product, you must state “This is
not an officially supported Google product” in an appropriate location such as
the project’s README file."

[https://opensource.google/docs/releasing/publishing/](https://opensource.google/docs/releasing/publishing/)

------
airstrike
I, for one, am ready for the metaverse

~~~
hprotagonist
do let me know when you’ve got a decent motorcycle model up and running.

------
pmayrgundter
The annotated SVG skeleton is a separate download, but that server is out of
quota :( Someone have it?

~~~
yohann305
you can find about 5 svg skeletons inside the github's folder:
resources/illustration/ Have fun

~~~
scruffups
Why is this down voted? I hate it when people do that and offer no
explanation. Totally useless input, even harmful. HN should force comment on
down-vote.

~~~
dang
Posting like this breaks the site guidelines. Would you please read them?
[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

One reason that guideline exists is that unfair downvotes frequently get
canceled by users who come along, see the situation, and make a corrective
upvote. Meanwhile complaints like this linger on in the thread, inaccurate and
off-topic—they don't garbage-collect themselves. As an example, I noticed your
other comment and upvoted it before I saw this comment here. Similarly, other
users have upvoted the GP.

As with any stochastic process, there is a lot of error and spillage with
downvotes. There's no way to perfect it; you have to ask whether the system is
better off with it than without it. Forcing comments wouldn't help, and
posting complaints certainly doesn't help.

~~~
scruffups
Why not have a meta discussion like a Wikipedia Talk page so comments like
these would have room and critique won't be shut down.

Yesterday (you can look in my comment history) I ran into a situation where
the person doing the down-voting turned out to be basing it on their opinion
not fact (after they finally stated their opinion on the matter, which
contradicts peer-reviewed research on the topic, I realized why they were down
voting: insufficient depth of understanding of the topic) and no one came
after them to correct the situation...

Either have people explain why they down-voted or have a Talk page where
people can discuss their reasons, complain, etc, behind the scene.

~~~
dang
HN is a site for intellectual curiosity
([https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20curiosity%20optimiz&sort=byDate&type=comment))
and a meta forum would be a step away from that. It would fill up with
bickering, nitpicking, litigating, and demands for bureaucratic
administration—all things that intellectually curious discussion requires
having the restraint to avoid.

------
terpimost
Great to see this thing open sourced. Adobe has a product for such things.

~~~
nine_k
At the level of fidelity of the demo, it's not yet a product.

The demo is very promising, though.

------
zanybear
I would really like this imposed on a Teams, Zoom, Lync, WebMeeting. This way
you cam still be in pajamas...

------
BIackSwan
This is freaking amazing. So many potential applications. Great work!

------
iseanstevens
This is great :) 10 years ago I'd been using Animata
[http://animata.kibu.hu](http://animata.kibu.hu) \+ kinect, this rolls it all
into one web based thing. cool stuff. Would it be easy to move it to a local
GPU accelerated version for more FPS?

~~~
soylentgraham
Hmm tfjs does use the gpu, and there are plenty of small models for posenet to
run fast on mobile. Maybe this is an old version. (Or maybe the bottleneck is
in the face or svg stuff) But to answer your question, yeah it could be faster
(everything could be faster given enough time! :)

------
ml_basics
This is great! It's amazing to see what can be made out of open sourced
machine learned building blocks

------
runawaybottle
Awesome, this is getting the idea mill in my head started. Animated short
comics would be a great use case.

Need some South Park vector models!

------
garfieldnate
IIRC there was a similar thing done for one of the VR systems recently, but
for the hands instead of for the face. Has anyone seen any open models or
software for tracking hand shape and location in this way? I would love to use
it in connection with sign language processing.

~~~
sethjgore
Yep! I do have something bookmarked somewhere. I use sign language myself and
do want the same thing. Curious about your background

~~~
garfieldnate
If you find that bookmark could you share it with me? One project I've looked
at is sign puppet [1]. It has all of the animation basics needed; the tough
thing is inputting the parameters to animate the puppet. Traditionally
capturing sign language data for the computer requires really sophisticated
tracking equipment (gloves, etc.). Being able to do it with a web cam could be
a game changer!

I studied linguistics and CS in school, and I learned a little JSL to speak
with deaf friends in Japan. I think sign language processing is a really neat
combination of computer vision/graphics and linguistics. Lately there have
been so many great advances in speech processing, but there hasn't been a huge
leap forward for sign language processing, though I feel there should be.

Deaf people are already really disadvantaged in many places, and getting left
behind technologically doesn't help. I really resented looking for JSL books
in the "disabled" section of the book stores in Japan, and when I spoke with
some people about JSL, they didn't believe it was its own language. Even just
linguistic work for sign languages is limited; I haven't seen a single
reference grammar (re: comprehensive documentation) on any sign language. I
think the difficulty of working with sign language data makes it more daunting
to work with. (Paucity of speakers is certainly not a deterrent for
linguists.)

[1] [https://github.com/aslfont/sign-puppet](https://github.com/aslfont/sign-
puppet)

------
mkchoi212
Wow really cool! I'm wondering how much work it is to define the specific
anchor/interest points on the SVG for the correct mapping to occur.

But I guess since "modern" illustrations are quite minimal, said work probably
shouldn't take too long.

------
nine_k
Next step: output to a deep-fake synthesizer instead of SVG. Jump from
animation to cinema.

------
kevmo314
Awesome stuff! I was just looking at face-api.js the other day, this kicks it
up a notch.

------
gitgud
Works okay for a browser application, but are there any native alternatives
for this?

------
saadalem
That's what a Side-Project should look like, curious to see the blog post !

------
ravila4
This is very cool! I wonder if using input from multiple cameras, plus a 3D
rig can improve the accuracy of the rendering.

~~~
Budabellly
Certainly, there are lots of apps leveraging the TrueDepth sensor on iPhones
that reach an even higher level of fidelity as far is pose estimation is
concerned.

The limiting factor for accuracy in a lot of these technologies is the actual
rigging process of the characters, probably because that is very difficult to
standardize or generalize across different geometries, art styles, animation
drivers, 2D vs 3D, etc.

------
mbalex99
Is this very similar to the tech that powers Adobe Character Animator?

------
tianshuo
Head-turning doesn't work that great, live2d looks much better here.

------
nitsky
This is awesome! Great work, thanks for sharing.

------
ghego1
This is incredibly awesome

------
marcsto2
This is awesome!

------
scruffups
Bravo!

