Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
numpad0
1 day ago
|
parent
|
context
|
favorite
| on:
Generative AI uses more energy on non-English text
Aren't tokenizers a hack to turn Latin alphabet texts into a pseudo Chinese with better per-unit information density?
help
brainwad
1 day ago
[–]
It is desirable for all alphabets/abjads/abugidas/syllabaries, not just Latin. Chinese is rather the special case of a human language with very high character inventory and hence low character/word ratio.
reply
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: