
SignAloud: Gloves That Translate Sign Language into Text and Speech - Mz
http://lemelson.mit.edu/winners/thomas-pryor-and-navid-azodi
======
skrap
To properly translate ASL, you need to pick up a significant amount of grammar
from the face (called non-manual markers) as well as body position, 3d capture
of spatial relationships between signs as they are made, and other features
which would be inaccessible to these sensors.

And then there's the fact that ASL isn't serialized English (it's actually
unrelated to English at all, and has its roots in the French Sign Language).
There's no one-to-one mapping between ASL signs and English words, just as
there's no one-to-one mapping between Chinese and French, or any two
languages. And ASL has grammar features not seen in any spoken language, to
complicate things further.

These sensors could perhaps be used to translate other sign languages, but I'm
really skeptical.

~~~
emodendroket
Yeah, even if they could perfectly transcribe the signs you'd have real
problems/limitations here I think.

~~~
gugagore
It's not really clear what it means to transcribe the signs anyway. There
isn't very great notation for signed languages

~~~
Zuider
I wasn't aware of the existence of any notation at all! The dictionary I have
uses photographs, other dictionaries and textbooks use drawings of people
speaking SL. The nearest thing to a notation in this case consists of a verbal
description of face and hand gestures as they are carried out in real time.

I have often wondered if it would be possible to create a notation using the
simplest and most abstract rendition of an actual SL speaker's signing. Lines
and curves that hold the semantic essence of the movement of face, hand and
posture. It would take a native speaker to be able to abstract what is truly
essential to meaning, and a community to agree on its elegance and clarity.

Off topic, but connected: I discovered from experience that small children can
form the SL alphabet with their hands before they acquire the coordination to
draw written letters. I found it very useful in teaching my children to read.

~~~
Zuider
Correction to my comment above: On taking out my Dictionary of British Sign
Language, I find that there is indeed an intricate notation used throughout.

------
AdamSC1
It's a great project, but to nail this they are going to need more involvement
from the deaf community as they've missed some key points:

1)ASL isn't just about hand movements, there is a ton of grammar that comes
from facial expression. It's not just like English tone, but can actually
change a word. Like can become "don't like" from facial expression.

2) Context is very key, there are a lot of words with multiple meanings that
matter based on the sentence. It could be figured out from a really smart
system but wouldn't be easy.

3) Lastly, ASL isn't English. This is the biggest mistake people make. You
aren't simply putting words into symbols. ASL has it's own grammar, syntax and
even slang. I remember spending 15 minutes trying to figure out what my ASL
teacher was saying when he said he "ate champ burger" and it turns out he
meant it was the best burger he ever had. To 'translate' ASL you need to move
from symbols to words and then add in the context.

Ultimately there is a reason that professionals in this industry are called
"sign language interpreters" and not "sign language translators" ASL is really
hard to put into spoken or written language.

It's also worth noting at really high level events an ASL interpreter will
translate spoken word into sign and then a "Certified Deaf Interpreter" will
re-sign it because the ASL of someone who is has communicated in sign their
entire life has far more facial expression and meaning than even professional
interpreters who are hearing.

I think it's great to see advancements in technology trying to assist the deaf
community, but I think it's important to get early involvement rather than
assume we know what people need.

I hope they keep up the great work - it's on the right path.

~~~
gugagore
Actually, the distinction (that I've seen) between "translator" and
"interpreter" is whether the "transformation" is happening in real-time across
spoken/sign language, or whether it's written language. Translators take text
in one language and write it in another, like translating a legal document.
Interpreters consume speech/sign in one language and speak/sign it in another.

------
ohitsdom
Seems cool, but I wonder how much of this is faked in the demo video. The man
signs "T" and it's translated to "Thomas", "N" to "Navid", and "L" to
"Lemelson". So unless this is a learning system as well, I'm skeptical. Lot of
potential here for sure though, hope they keep it up.

~~~
ToastyMallows
I wonder if they've just hard coded their names for now because they had
trouble doing fingerspelling, because I doubt that someone's name-sign is just
a 'T'. That's really close to the sign for 'bathroom', which would be weird :)

~~~
pavel_lishin
Could be a home-sign, or a common shortcut. If two people are discussing two
people who aren't physically present, using just their first initials could be
a very obvious way of referring to them unambiguously without having to sign
their full names.

~~~
Zuider
Agree. It depends on context. A quick flash of 't' could also mean 'thanks',
or 'th' if it was less obvious.

~~~
gugagore
Have you seen that, or are you speculating?

~~~
Zuider
I have experienced this in communication with deaf colleagues.

~~~
gugagore
Can you describe why/how this would be used instead of the dictionary sign for
THANKS or GOOD?

~~~
Zuider
Here are the why/hows as I understand them:

1) This is an informal shorthand agreed between people who already know each
other in the particular context of work, so there is no surprise or ambiguity.
Outside this context, I would use, and expect to see the dictionary sign.

2) It functions as an 'in-group' slang, boosting cohesion and good will.

3) There is a certain humorousness in bending a symbol into an informal use.
This is more pronounced in cases where the alternate meaning is irreverent or
vulgar. I have no intention of expanding on the second point.

4) One does not have to be facing the SL speaker. The 't' symbol can be
recognized from beside the person or even behind. Point 5 gives the rationale
for this.

5) It can be communicated without changing focus or posture, allowing
concentration on the task at hand without breaking the workflow.

6) Though 'thanks' is a brief gesture, 't' can be signed and understood even
more quickly. Just a flash of meaning.

7) The sign for 'good', while it is also used to communicate thanks and has
advantages 4,5,6 listed above, can be ambiguous, violating the principle of
least surprise (see point 1 above). It is more generic, while 't' is specific.

~~~
gugagore
Interesting. I'll ask around about that.

However, I think it's fair to say that a rudimentary system like this one is
scripted, and is not tuned to the informal and complicated contexts you
describe. You can argue that it's fair game to pre-program the device to emit
"Lemelson" for #L, just as you might prepare an interpreter with proper nouns
before a gig, but I think it goes against the spirit of the claims made in the
video.

------
fareesh
This is a really novel approach, but IMHO a more well rounded solution is the
one being developed at Microsoft Research using Kinect.
[https://www.youtube.com/watch?v=HnkQyUo3134](https://www.youtube.com/watch?v=HnkQyUo3134)

Nonetheless, great work by these students. I think their work is likely to
have more applications in other areas where mimicking the movement of hands
and fingers is an essential component.

------
xfour
Congo was way ahead of its time. Amy good Gorilla!
[http://m.youtube.com/watch?v=KXlSFKLuzTA](http://m.youtube.com/watch?v=KXlSFKLuzTA)

~~~
c3534l
There hasn't been a better movie with laser-mounted gorillas since.

------
fsiefken
So now you can text entry without using a keyboard or your voice, good for rsi
and good for faster entry, gesture based computing.

~~~
sleepychu
Learning sign language as an adult and using it a lot can give you really
terrible RSI.

~~~
drewbug01
If you're getting RSI from sign language you're either an interpreter or over-
producing.

In the case of the former, you're using the language differently than a
typical user. Most people don't produce language in the same amounts or in the
same ways that an interpreter will (that is; speaking/signing isn't most
people's job). There are steps you can take to reduce RSI as an interpreter,
including proper rest while working and supportive braces at night.

For the latter, it's possible to over-produce signs - basically "signing too
hard." That can be painful, but it's correctable.

~~~
jallmann
> Most people don't produce language in the same amounts that an interpreter
> will

It isn't really because of how much they sign (deaf people don't typically get
RSI from sign language), but rather because of _how_ they sign it. The signing
is more precise and formal, the hands/arms are elevated more, and generally
less "colloquial" than you would see in day-to-day sign language.

~~~
drewbug01
Volume of signing isn't totally unrelated - even if you presume 4 hours of
signing time per day, most people don't speak/sign for that long an amount of
time. It's more than a typical deaf person, any way you slice it.

But, you're right - the manner of signing has something to do with it as well,
and I've amended my comment to reflect that. :)

------
stevenbedrick
A useful commentary from a linguist who studies ASL:

[http://katies.online/katiesblog/index.php/2016/04/27/yes-
ive...](http://katies.online/katiesblog/index.php/2016/04/27/yes-ive-seen-the-
signing-gloves/)

------
luxpir
Very cool, but it's not quite the revolution described by the inventors in the
video - isn't it only one-way, assuming a lot of the deaf person's lip-reading
ability?

~~~
emodendroket
If a deaf person can read lips they can probably also speak English.

~~~
luxpir
I'm not sure we understand each other. If the signing person can speak to
employers etc. now via the device, they can only speak back with their mouths,
or possibly speech recognition if well set up. The deaf person would have to
be able to lip-read people who potentially don't know how to help them in that
area. In other words, it might still be tricky to communicate.

~~~
emodendroket
You are right; I totally misunderstood you. Sorry.

------
pvinis
some friends of mine translate sign language to text/speech using leap motion
and/or kinect to figure out the best way for it. there is a startup that uses
leap motion and an ipad but it looks very clunky to use. my friends are
connecting to the user's smartphone for the translation work.

------
breezest
I heard a similar work done by the students in Chinese University university
of Hong Kong. But they make their own gloves, which cost about $75.
[https://translate.google.com/translate?sl=zh-
CN&tl=en&js=y&p...](https://translate.google.com/translate?sl=zh-
CN&tl=en&js=y&prev=_t&hl=zh-
TW&ie=UTF-8&u=https%3A%2F%2Fwww.cse.cuhk.edu.hk%2Fv7%2Fb5%2Fabout%2Fachieve%2F84.html&edit-
text=)

------
zardeh
I always find these cool, and then remember that I know someone who built a
working (well working!) prototype of this in a weekend no almost no fanfare,
from practically scratch.

~~~
avs733
with the caveat of not knowing your that someone in any way shape or form...

These are students, young/new engineers. This is impressive and should be
lauded as should all attempts to push what an individual person is capable of.
It may not be new or novel to everyone, but it is to them...basically, cut
them a bit of slack.

~~~
zardeh
The people I'm talking about were also students, (University Hackathon). A
group of four, I want to say 2 sophomores and 2 freshmen.

This was 2-3 years ago, so machine learning wasn't advanced (and even now,
getting a functional NN running and trained in a weekend isn't feasible) so
they only had it recognize a small set of commands, but it could do that.

------
scholia
There's a longer and more informative story at UW Today:

[http://www.washington.edu/news/2016/04/12/uw-
undergraduate-t...](http://www.washington.edu/news/2016/04/12/uw-
undergraduate-team-wins-10000-lemelson-mit-student-prize-for-gloves-that-
translate-sign-language/)

------
codingdave
With all the critiques in the comments so far, it is worth noting that this
article simply says they won a contest for being promising inventors, not that
the invention doesn't have flaws.

~~~
gugagore
Did you watch the corresponding video

------
buraksarica
Another example:
[http://www.turtela.com/en/main](http://www.turtela.com/en/main)

------
macu
I would love a new text entry method using gloves or finger/toe trackers. For
recording thoughts while on the go.

~~~
scholia
Have you tried a chorded keyboard?

[https://en.wikipedia.org/wiki/Chorded_keyboard](https://en.wikipedia.org/wiki/Chorded_keyboard)

------
gloves
I don't translate sign language into text and speech!

~~~
Zuider
Don't know why you got downvoted. Your comment is both admirably terse, and
completely valid!

SL is an entirely distinct language. People who are fluent in multiple
languages switch from linguistic context to another when they speak different
languages. They don't translate from their native language. So it makes sense
that you don't do this either.

