On: Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference

[meowface]

Thank you for the breakdown. Bit of an emotional journey.

"$500 in the future...? Oh, $30 million now, so that might be a while..."

[jamalaramala → meowface]

It took 30 years for computers go from entire rooms to desktops, and another 30 years to go from desktops to our pockets.

I don't know if we can extrapolate, but I can imagine AI inference on our desktops for $500 in a few years...

[stefs → jamalaramala]

well, we can AI inference on our desktops for $500 today, just with smaller models and far slower.

[ryao → stefs]

There is no need to use smaller models. You can run the biggest models such as llama 3.1 405B on a fairly low end desktop today:

https://github.com/lyogavin/airllm

However, it will be far slower as you said.