
Getting Alexa to Respond to Sign Language Using Your Webcam and TensorFlow.js - hardmaru
https://medium.com/tensorflow/getting-alexa-to-respond-to-sign-language-using-your-webcam-and-tensorflow-js-735ccc1e6d3f
======
imglorp
ASL student here. I just wanted to highlight how many challenges there are to
translating, even for humans. Here's two examples.

First is dialect. One would think the Internet/youtube/videochat would
vaporize geography differences, and it's starting to happen, but that's not
(yet) the case. Lots of signs have "synonyms" based on signer's preference,
etymology, who they learned from and who they hang out with, etc.

Another is grammar. What OP is doing in his video, for example is English
grammar, where signs are directly substitited for words. But ASL has its own
sign order, modifiers like facial expressions, and idioms for brevity. For
example "have you ever been to San Francisco before?" might be "SF TOUCH YOU
FINISH" with raised eyebrows at the end, as a modifier. Note also there are no
conjugations or articles like some languages, but there are pronouns. In fact,
there are local bindings where you make up a sign name for someone on the fly
and then use it during a conversation.

I think with large enough training sets, this will all be mitigated, like
Google needed years of speech samples to get Translate working okay.

~~~
mabbo
I took 2.5 years of ASL in University, and I've always been annoyed by this,
but had a hard time describing what I mean to others. You have just summed up
the problem perfectly in a few sentences, and I thank you for that.

That said, I think the OP is still on the right track here:

> I put it together so you can train it on your own set of word and
> sign/gesture combos.

A Deaf person should be able to train the system on each command they want so
that it works for them automatically in the dialect they want.

> there are local bindings where you make up a sign name for someone on the
> fly and then use it during a conversation

My very favourite thing about ASL, having learned programming beforehand, was
that I could assign people to variables/registers in space. "John _point at
spot to my left_ said to Susan _spot on right_ bla bla bla" and then later be
able to just point at my left and everyone knows I mean John. And then if John
moves to the right, I've just moved him into that register and can reassign
the left one to someone or something else. And with a group of experienced
Signers, everyone just comprehends this perfectly.

~~~
imglorp
Ah, pointing at sign space for someone... I like to think of that like a
pronoun (there's probably an official term).

There's also this thing where you introduce a character, then make up a
"temporary name sign" on the fly. Don't know what that's called either :)

~~~
mercurywells
The term is 'deixis'. Not sure about the latter.

------
hardmaru
GitHub: [https://github.com/shekit/alexa-sign-language-
translator](https://github.com/shekit/alexa-sign-language-translator)

Web Demo: [https://shekit.github.io/alexa-sign-language-
translator/](https://shekit.github.io/alexa-sign-language-translator/)

------
abhi3188
Hey, I'm the guy who made this! Happy to answer any questions that the blog
didn't address

~~~
hammock
Do you think you could get this to work for Google Assistant as well?

Does ASL-to-text already exist outside of voice assistants, or is that part
totally new?

------
tango24
Whoa, this is really cool! My wife knows sign language and I showed her this
and she was impressed, and had so many questions about how it worked =) I love
the innovation — Thanks for sharing!

------
pbhjpbhj
Interesting I think if I were approaching BSL sign interpretation I'd start
with recognising hand shapes, and then move on to other elements of sign
notation ... in theory then you could feed the system a dictionary and
recognise all signs.

I think BSL uses a variant of Stokoe notation.

Presumably the system of the OP could be modified to recognise
dance/skating/snowboarding/martial arts moves.

It seems that once you have an ability to model every aspect of body position
from video (as there doing for enhanced love-action sequences in movies now)
that this sort of thing becomes much easier.

------
jakamau
This looks interesting. I'd be interested to see how the set-up works when it
is trained with someone that signs natively.

Most of my experience in the deaf community has been at trivia nights where
the signing is incredibly fast (to me) and grammatically loose (just like
everyone that's voicing at the bar).

In the end I think it's an invaluable avenue to explore, the tech is important
no matter what the end device looks like. Great work.

------
al_ramich
Love the concept, great work. Came across these guys who are working on making
travel more accessible for the deaf community. I would see some overlap in
using Alexa to make more voice inclusive content accessible

[https://startupyard.com/meet-deaftravel-building-a-better-
in...](https://startupyard.com/meet-deaftravel-building-a-better-internet-for-
deaf-people/)

------
cwt137
This is similar to an AWS DeepLens community project called ASLens
[https://aws.amazon.com/deeplens/community-
projects/ASLens/](https://aws.amazon.com/deeplens/community-projects/ASLens/)

------
ai_ia
Although off topic, I loved your paper with Schmidhuber, hardmaru. :)

------
neha
hi

------
ausjke
How many are keeping using alexa/echo these days? mine were turned off months
ago and never felt the need to switch them on again, phone does pretty much
everything I need, actually it does more than what I need.

plus I don't feel comfortable that there is a silent listening device lurking
24x7 for all family members, so it is not just they're useless, but I
intentionally do not want to have them on at all 99% of the time.

the only amazon device I use is FireTV for netflix and Amazon prime video, all
amazon ereaders, echo, alexa are replaced by my cellphone...

to that extent, i feel amazon stock price is too high, too much hype for the
alexa/echo/kindles at least in my opinion

~~~
jonknee
Every day I am at home for weather, timers, lists.

~~~
ausjke
and phone does it all, plus being portable on-the-go

~~~
jonknee
Yes and we even did all these things before a phone and life was fine. The
question wasn't if I use the Echo to do things that are impossible otherwise.

I cook a lot and I find it a lot easier to just say "Alexa set a chicken timer
for 15 minutes" or "Alexa add black pepper" than it is to get out my phone and
do those things.

~~~
celestialjeu
agreed -- being able to verbally set timers is very valuable to me. Also I
have Lifx bulbs in my living room and it is much easier to tell Alexa to turn
them on/off/change brightness than it is to get out my phone. I only use the
phone app for that when I want to set up some custom color thing (which is
pretty rare tbh)

