Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

To be clear, this is a 7B model. It's just trained on 1.7 trillion tokens. At first I was confused why they were making such a big deal of of a massive 1.7T model outperforming a 7B model.

By the way, GPT-4 has been predicted to be using a 1.7T model, although OpenAI has not confirmed this to my knowledge.



The most interesting bit is that this is RWKV model, meaning constant size state (no quadratic attention). AFAIK the biggest open weights non-transformer model.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: