On: Chiplet ASIC supercomputers for LLMs like GPT-4

[brucethemoose2]

... Isn't this basically the Cerebras WS2? Each "die" has 40GB of SRAM, and they have a fast interconnect.

[punkgenius → brucethemoose2]

This seems to be more achievable and cost-effective than Cerebras. Some comments mention Cerebras cost millions for each 'die'.

[brucethemoose2 → punkgenius]

Yes, that because Cerebras's "chip" is actially an entire wafer many GPUs would normally be carved out of.

The extra stuff TSMC must do to pull that off are probably expensive... But I can't imagine it being, say, 10x more expensive than a wafer full of reticle sized dies (like Nvidia does). And thats setting aside the massive IO advantage of Cerebras's mega die.