Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

And yet nobody is doing that… big-token conspiracy
 help



"nobody is disabling this optimization" shouldn't be surprising. It's a matter of knowing what letters are in words versus a 4x token efficiency boost. Nobody cares enough about spelling.

If it was top priority, every company that can't find a post training fix would go disable half their tokenizer code and it would be solved in the next model.




Consider applying for YC's Winter 2027 batch! Applications are open till November 2.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: