Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Only libucidata is big.

Reason β€” tons of data. Contrary to what other commenters said, it’s not (only) Unicode properties (they are tiny), but a lot more: from rules to spelling out the numerics, to transliteration, collation, to word/sentence segmentation (some of which are absolutely non-trivial and sometimes require dictionaries of special cases).



> word/sentence segmentation (some of which are absolutely non-trivial and sometimes require dictionaries of special cases)

Not just special cases: for Thai, you need a dictionary to do segmentation at all.




Consider applying for YC's Winter 2027 batch! Applications are open till November 2.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: