
Real-Time Recognition of Handwritten Chinese Characters - aeontech
https://machinelearning.apple.com/2017/09/12/handwriting.html
======
FabHK
slightly off-topic, but the images in the article illustrate it beautifully: I
think many people don't realise (I didn't) what an amazing skill it is to be
able to read handwriting (cursive). I took it for granted to be able to read
hand-written German, English, Spanish, and totally forgot that it must have
taken me some time as a kid to acquire that skill (beyond reading and writing
generally).

However, when I studied Chinese for a bit, the (printed) characters are tricky
to start with, but reading a fluent writer's handwriting ("unconstrained", as
they call it in the article) is nigh impossible.

I think stroke order is hugely important, in that it forms the conventions for
fast abbreviated writing. For example, the "mouth" radical, a square, "口",
when written quickly basically looks like a lower-case "b", because the left
lower stroke comes first, then top and right in one stroke, then the bottom
stroke.

More complicated characters basically become a squiggle, and it's really
magical, if you think about it, that you can just read it. I mean, look at
figures 5, 6, 7 in that article.

For me, a nice illustration of Moravec's paradox of AI [1]: what we think is
difficult (chess, differentiation, ...) is easy, and what we think is easy
(reading handwriting, putting an egg in the fridge, walking (!) and opening a
door [2]) is hard...

[1]
[https://en.wikipedia.org/wiki/Moravec%27s_paradox](https://en.wikipedia.org/wiki/Moravec%27s_paradox)

[2]
[https://www.youtube.com/watch?v=g0TaYhjpOfo](https://www.youtube.com/watch?v=g0TaYhjpOfo)
("A Compilation of Robots Falling Down at the [2015] DARPA Robotics
Challenge")

~~~
imron
> but reading a fluent writer's handwriting ("unconstrained", as they call it
> in the article) is nigh impossible.

It is when you are starting out, but it's not so bad once you get used to it.
The best way to learn to read Chinese handwriting is to learn to write Chinese
handwriting, and I don't just mean practising writing by hand, but to work
through a textbook aimed at teaching cursive Chinese.

This book is a good introduction in English [0], and if you are comfortable
with Chinese only, this book is excellent and also far more comprehensive [1].

Once you've gone through those books, you'll be able to read most handwritten
stuff without too much struggle.

0: [https://www.amazon.com/Chinese-Cursive-Script-
Introduction-P...](https://www.amazon.com/Chinese-Cursive-Script-Introduction-
Publications/dp/0887100333)

1:
[https://www.amazon.cn/图书/dp/B00C907XM2](https://www.amazon.cn/图书/dp/B00C907XM2)

~~~
vacri
In the examples in the article, do you find the second example for 的 legible
(figure 5)? Or either of the characters in figure 7 - without being primed,
would you know what characters they were meant to be?

I once worked with a paediatric neurologist whose handwriting was so bad, even
he couldn't read it. We had a patient with a referral from him which we
couldn't read, took it down the hall to him... he scrunched up his face... and
gave up reading it: "Well, I know the patient, and they need...". Absolutely
lovely bloke, but I'd hate to get a referral from him without being able to
talk to him :)

~~~
imron
> do you find the second example for 的 legible (figure 5)?

Yes. All of the 的 examples are clearly 的 that I'd get without being primed.

Figure 7 is more tricky but I'd know 五 without being primed, and 王 definitely
after being primed _or_ if I was reading handwriting that had more context.

~~~
vacri
Thanks for the info. I've been learning now for about a year, and text is
definitely my weak spot. I see handwritten chinese and it frequently looks
nothing like the nice, crisp computer fonts (which are hard enough just by
themselves) - not so much that it's messy, but that I often just simply can't
see how you make out the elements. I find it amazing that not only do native
chinese speakers recognise thousands of characters, but also wild handwritten
variants, _and_ also the same again in simplified/traditional.

I once lightheartedly asked my teacher if exercise books in china had bigger
gaps between the lines, as I couldn't see how to fit a vertically-complex
hanzi in. She wandered over and scrawled something that neatly fit in the
line. "How the hell did you do that... ?" :)

~~~
imron
> Thanks for the info. I've been learning now for about a year,

A year is probably still too early to be able to read handwriting well. You
probably need to have about 2,000 known characters under your belt _and_ also
have a good feel for the language such that you are able to fill in gaps based
on context - that is be able to complete parts of the sentence based on
knowledge of what words/characters are likely to fit.

There are many cursive characters that reduce to something quite similar so
having knowledge of what fits in the sentence makes things easier.

As an example, with 王 above, by itself it might not look like anything you can
recognise, but if it was written like that on a note starting out like 尊敬的王老师
then from context it narrows down the possibilities significantly.

> She wandered over and scrawled something that neatly fit in the line. "How
> the hell did you do that... ?" :)

Practice! It's one thing to know how to write a character, it's another thing
to have the fine motor skills required to write it proportionally within a
confined space, and you can only develop that skill through practice. If
you're interested, one of my side projects [0] is a site that lets you
generate custom Chinese worksheets of varying grid sizes and fonts (including
some handwriting fonts) and other things, which can be useful for such
practice.

0: [https://www.hanzigrids.com/](https://www.hanzigrids.com/)

~~~
vacri
A year is definitely too early - I'm a middling student at it (and weak at
text), and would be lucky to get 100 characters by sight. As you say though,
it's all about practice, and I need to apply some "bum glue" (to quote Stephen
King) and just sit down and do it. A Chinese colleague of mine said that
throughout high school, he had nightly homework of 6 new characters, pretty
much every night. Ouch!

Thanks for the link - it will help and I've been meaning to look for one
(phone apps with your finger aren't quite the same).

~~~
imron
> A Chinese colleague of mine said that throughout high school, he had nightly
> homework of 6 new characters, pretty much every night. Ouch!

'Pretty much every night' is the most important part.

Learning Chinese characters is a grind and you just need to keep chipping away
at it.

Pick a nice sustainable number of characters (maybe 6, maybe 3, maybe 10 but
something that takes maybe 30 minutes to an hour to do including revision) and
then do it every day without fail.

Number of days in a row spent learning is far more important than number of
new words learnt per day, so always try to optimise for the former rather than
the latter.

At some point, almost every Chinese learner who has spent more than 3 years
learning the language has looked back and thought 'why oh why didn't I just
learn 3 characters a day'.

------
peterburkimsher
I'm very interested in this field! But I didn't stop at 30,000 characters -
I'm making a dataset of 75,000 characters from 1500 different fonts. Does
anyone want me to publish it?

I use Apple's Chinese handwriting keyboard almost every day, living in Taiwan.
Unfortunately it only shows 4 possible choices, and often the character I need
is not one of those. So I use my own Pingtype keyboard.

[https://pingtype.github.io](https://pingtype.github.io)

Enter a character into the bottom text field, click Search, and see the
character decomposition above, and the characters & words containing that
character below. Then when you're ready, click Insert to add it to the text
field at the top for translation.

I have no idea how to do marketing for this tool, and commenting here is the
only way I've had ~1300 views in the last 6 months since I published it.

~~~
lathiat
Sounds like you should apply for a Job at Apple, or sell your "company" :-)

------
rootbear
Over twenty years ago, I was in a conversation with a developer who was
working on Japanese handwriting recognition for the Go PenPoint OS. He told me
that stroke order, which is supposed to be consistent in Japanese writing, was
a big help. It's interesting that this article says that there was a need to
"achieve stroke-order independence". I'm wondering what the issues are here
and why stroke order was useful in the 90s but not considered so now. I'm also
not sure if the developer I spoke to was concerned with Chinese input as well,
or just Japanese, and how that might affect the methods used.

~~~
tobltobs
With handwriting recognition you differ between online and offline
recognition. With online you have the stroke order, which helps a lot for
character segmentation. Character segmentation is the unsolved problem for
handwriting recognition, at least for Latin scripts. With offline you only
have the image of a word. If you depend on online information you can't
recognize text from images taken with a camera.

~~~
barbolo
Segmentation is not needed for a modern deep learning system, since this is
learned by the neural network. It’s a solved problem for many handwritting
recognition problems.

~~~
tobltobs
Would love to see a system or a paper which did solve this problem.

------
Isamu
Hey, I understand this! Because I read An Intuitive Explanation of CNN's
yesterday!
([https://news.ycombinator.com/item?id=15328516](https://news.ycombinator.com/item?id=15328516))

Thanks HN!

------
Scaevolus
The input is a 48x48 image. For handwriting recognition, many systems use
stroke information (online) to augment the purely image based (offline) data.
This significantly reduces error rates.

For the smaller ICDAR 2013 Chinese Handwriting Recognition Competition (~8,000
characters), CR(1) accuracy went from 94.77% (offline) to 97.39% (online).
Human recognition rates were 95.19% (offline) and 96.13% (online).

~~~
raverbashing
Yes, but the article mentions there's a lot of stroke order inconsistency
(errors), so they don't rely on that

~~~
taneq
Surely you could still use it for hints? (ie. "stroke X followed by stroke Y
has a Z% higher chance of being this character rather than that one", much
like some phone keyboards use the prior/next words entered to correct words so
they make sense in context).

~~~
raverbashing
Problem is that when these hints push you in the wrong direction

One of the examples there was a Chinese character being drawn as something
like a 2. The character looks nothing like a 2

------
soared
I was going to ask if anyone could pinpoint when apple started publishing, but
I found their first post - July 2017. They even have an email inviting
students, researchers, and devs to ask questions.

[https://machinelearning.apple.com/2017/07/01/Welcome.html](https://machinelearning.apple.com/2017/07/01/Welcome.html)

------
Steko
I applaud 'Handwriting Recognition Team' on the excellent work they're doing.

------
wibr
Unfortunately they don't provide any details about the model or the training.
Would be nice to have an example as a baseline for real-time performance on
iOS devices.

------
ww520
What would be really cool is to have an augmented VR setup where the user is
writing in midair and the VR shows light trace of it, while the OCR
recognizing the writing.

------
ocrimgproc
Has anyone done any work on English handwriting recognition? I'm very
interested in this field.

~~~
rabboRubble
Chinese language student, native English speaker. I haven't found any
handwriting recognition commercially available for the English language that
compares to what is available for Chinese Traditional / Simplified.

The best English handwriting recognition software I've found is the old Palm
Pilot Graffiti writing system. It's not really developed since the late 90's /
00s. Back in the day, it was blazing fast (and pretty accurate after learning
the method) compared to shitty small mechanical keyboards of the era.

After diving into Chinese I realized how deficient the English HR landscape
was and how badly under-researched the area is. I think the standard QWERTY
keyboard is just too efficient for the Euro zone languages to make the
research investment worthwhile. My 2 cents / guess.

There are Graffiti like apps available on IOS. What I've found is that with
english keyboard auto-correct, my typing is so much faster without Graffiti
using the standard Apple IOS English keyboard. Kind of a disappointment...

Another conjecture, I suspect much of the morbidity in English HR research has
something to do with Graffiti patent lock up between what ever remains of
Palm, Xerox, and some heretofore unheard of company called Communication
Intelligence Corporation.

~~~
tonfa
On android,
[https://play.google.com/store/apps/details?id=com.google.and...](https://play.google.com/store/apps/details?id=com.google.android.apps.handwriting.ime&hl=en)
supports a bunch of languages (and emoji).

~~~
rabboRubble
I hated the Android phone I had 5 years ago and haven't looked back, so
apparently I'm not up to speed on developments in the Android marketplace.

I stand corrected on everything stated above!

I'll have to borrow a friend's phone to play with this and see how it compares
to the old Graffiti.

------
DelightOne
Is there an API for this or is this for Scribble/Apple Apps only?

~~~
paxswill
I think this is only exposed in iOS with the Chinese keyboards (and Scribble
on watchOS), so not programmatically.

~~~
peterburkimsher
Yes, and also on Mac OS. (Show Trackpad Handwriting)

