I think you're right that demand for inference is likely to grow by orders of magnitude in the medium term, but I don't know that Nvidia's lead over their competitors is likely to remain as strong as it is today. That is, they'll probably hold their dominant position, especially for training, but I'd expect their lead on inference to shrink by a lot. Companies pretty much understand how to make inference engines. Expect AMD, Intel, Qualcomm, Apple and others to come out with products geared at fast and efficient inference. Not to mention Grok and other dedicated chips.
Microsoft just announced that they are getting better price/performance ratios with AMD for inference. It's happening already. Most of GPU thirst for training at the moment but eventually most of workload will be inference. And that's depending on a big if, if users are willing to pay for LLM generated content as much as we're hyped about it right now.
I used to think this. I still mostly think this (and have frankly lost a lot of money in opportunity cost because I thought it), but I've been astonished at how long AMD is taking to pivot, such that there are startup competitors like Groq that might honestly beat them out.
CUDA is a technology investment that gives Nvidia a moat, but there is nothing exclusionary about it, Nvidia invested in tooling years ago ... it will just take time for others to catch up.
NVDA banned CUDA translations layers, that's pretty exclusionary because it denies interoperability. There was already a discussion about it here: https://news.ycombinator.com/item?id=39592689
Well, they tried to legally prohibit it, but as people in that thread pointed out, it's legally irrelevant in most developed countries ... so another hardware maker can fund a (legally) independent team in say France (most favorable reverse engineering laws) to break it / copy it / do whatever they like ... and then open source the output for the rest of the world.
It's even dubiously legal in the US ... but there it would be a war of legal fund attrition and Nvidia has deep pockets.
Still - break it in France, give it away ... US laws go poof.
It's really just the ongoing development of the software ecosystem that gives them their moat ... but it's a good technical moat because they steer future dev and others play catch up.
