Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Forgive the ignorance, but how are you running a 763B parameter model on 64GB of RAM?


SSD weights streaming. The actual weights is 552B, then engram.


which engine?



By “quantitizing” it


The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: