On: The AMD Radeon Instinct MI300A's Giant Memory Subsystem

[DiabloD3]

Funny you say that, because nobody serious about AI is actually using Nvidia unless they're already locked in with CUDA.

Highest performing inference engines all use Vulkan, and are either faster per dollarwatt on the CDNA3 cards or (surprisingly) the RDNA3 cards, not Lovelace.

[behnamoh → DiabloD3]

> Funny you say that, because nobody serious about AI is actually using Nvidia unless they're already locked in with CUDA.

Yeah right, so Meta and XAI buying hundreds of Nvidia's H100's was because they were not serious in AI. wtf

[DiabloD3 → behnamoh]

Meta has an in-house accelerator that the Triton inference engine supports (which they use almost exclusively for their fake content/fake profiles project). Triton is legacy software and, afaik, does not have a Vulcan backend, so Meta may be locked out of better options until it does.

That doesn't stop Meta's Llama family of models running on anything and everything _outside_ of Meta, though. Llama.cpp works on everything, for example, but Meta doesn't use it.