We've been hearing that line for years, but HN comments are a valuable source of info on where NVDA might go in future. You're the only poster here I could name who is strongly pro-AMD in the AI space. Everyone else seems to put their toes in the water in the hope of being able to get an edge by avoiding NVIDIA's monopoly prices, immediately gets burned by trash software quality and runs away screaming "never again".
I only dabble in AI stuff but have decades of experience doing quick surface-level quality checks of open source projects. I looked at some of AMD's ROCm repos late last year. Even basic stuff like the documentation for their RNG libraries didn't inspire confidence. READMEs had blatant typos in, everything gave off a feeling of immense lack of effort or care. Looking again today the ROCrand docs do seem improved, at least on the surface, I haven't tried it out for real.
But if we cast the net a little wider again, the same problems rear their ugly head. Flash Attention is a pretty important kernel to have if working with LLMs, maybe I'd like one of those for AMD hardware?
We're in luck! An official AMD repo with flash attention in it, great! Except.... the README says at the top:
Requirements: CUDA 11.4 and above. We recommend the Pytorch container from Nvidia, which has all the required tools to install FlashAttention.
Really? Ah, if we scroll down all the way to the bottom we can find a new section that says "AMD/ROCm: Prerequisite: MI200 & MI300 GPUs". Guys, why not just rewrite the README, literally the first thing you see, to put the most important information up front? Why not ensure it makes sense? It takes 10 seconds and is the kind of attention to detail that makes me think the rest of your work will be high quality too.
Checking the issue tracker we see people reporting that the fork is very out of date, and that some models just mysteriously don't work with it due to bugs. These issue reports go unanswered for months. And let's not even go there on the hardware compatibility front, everyone already knows what "AMD support" really means (not the AMD cards you might actually own) vs what "NVIDIA support" means (any device that supports the needed CUDA version, of any size).
I would never try to defend AMD with regards to them needing to catch up. Even talking with executives at AMD, neither would they. Nobody is trying to pull a fast one on this.
What has changed for certain, is their attitude and attention. I just got back from Dell Tech World. Dell was caught off-guard with this AI thing too. It is obvious the only thing that anyone is talking about now is "ai ai ai ai ai ai".
Give them a bit of time and I think they will start to become competitive over the next few years. It won't happen over night. You won't see README's fixed right away. But one thing that is for certain, they are all at least trying now, instead of pretending it doesn't exist.
Whether they will be successful or not, is yet to be seen. I wouldn't even know how to define successful. I don't think anyone is kidding themselves about Nvidia being dominant. But, I'm personally willing to bet on them selling a lot of hardware and working on their software story.
It's great that you're pushing AMD forward, no doubt about it! And I'm sure they'll make good progress now. Like I said, the ROCrand repository seems to be in much better shape now.
What is telling for me is that ROCm itself is on regular cadence updates. Not just small updates, but actual meaningful fixes and improvements.
Not only that, but it is all being done in the open, unlike their competition. Hotz demanded some documentation, they provided it and he still complained. Some people just can't find happiness.
Now, whether or not I am pushing them forward is yet to be seen, but at least I'm trying. By positioning myself as a new startup who's trying to help... that will easily garner all their support as well. As I said in another comment, why not let them try too?
I only dabble in AI stuff but have decades of experience doing quick surface-level quality checks of open source projects. I looked at some of AMD's ROCm repos late last year. Even basic stuff like the documentation for their RNG libraries didn't inspire confidence. READMEs had blatant typos in, everything gave off a feeling of immense lack of effort or care. Looking again today the ROCrand docs do seem improved, at least on the surface, I haven't tried it out for real.
But if we cast the net a little wider again, the same problems rear their ugly head. Flash Attention is a pretty important kernel to have if working with LLMs, maybe I'd like one of those for AMD hardware?
https://github.com/ROCm/flash-attention
We're in luck! An official AMD repo with flash attention in it, great! Except.... the README says at the top:
Requirements: CUDA 11.4 and above. We recommend the Pytorch container from Nvidia, which has all the required tools to install FlashAttention.
Really? Ah, if we scroll down all the way to the bottom we can find a new section that says "AMD/ROCm: Prerequisite: MI200 & MI300 GPUs". Guys, why not just rewrite the README, literally the first thing you see, to put the most important information up front? Why not ensure it makes sense? It takes 10 seconds and is the kind of attention to detail that makes me think the rest of your work will be high quality too.
Checking the issue tracker we see people reporting that the fork is very out of date, and that some models just mysteriously don't work with it due to bugs. These issue reports go unanswered for months. And let's not even go there on the hardware compatibility front, everyone already knows what "AMD support" really means (not the AMD cards you might actually own) vs what "NVIDIA support" means (any device that supports the needed CUDA version, of any size).