Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Surely that can be tried! of course at a small scale if you're short on GPUs. what I understand is that V4 replaces MLA with a hybrid of two compressed attention types and changes the residual connections, while keeping the mixture-of-experts design, but I haven't gone in-depth architecture :)
 help



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: