Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Aren't tokenizers a hack to turn Latin alphabet texts into a pseudo Chinese with better per-unit information density?
 help



It is desirable for all alphabets/abjads/abugidas/syllabaries, not just Latin. Chinese is rather the special case of a human language with very high character inventory and hence low character/word ratio.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: