> You can run Stable Diffusion and LLaMA locally on a MacBook Pro (with Automatic1111 and llama.cpp
Apple themselves directly ported stable diffusion to metal long ago. You can run LLaMA through Vulkan or Metal with Apache TVM, on an iPhone GPU, though this is admitedly very new.
Honestly, one of the biggest issues in AI land is discoverability and the snowball effect. People tend to congregate around the Nvidia focused, pytorch eager mode projects, which sucks all the oxygen out of the room and leaves very interesting ports to languish.
Yeah, inferencing capabilities is small-fries. If you have an ARMv8 multicore system, you can leverage ARMNN acceleration for a variety of AI tasks. Intel CPUs have OpenVINO and AVX2, and Ryzen systems can at least get some of the SIMD optimizations. I can run Stable Diffusion and LLaMA on my Broadcom SOC, if I wanted.
The biggest issues (imo) are centered around the runtime and compute capabilities. CUDA is a software juggernaut, and simply stepping up to the plate would require an unprecedented investment on Apple's part. It might be impossible anyways, CUDA supports more CPU arches than MacOS does and goes all-in on containerization tech that the current XNU kernel simply doesn't work with. And then there's the performance problem - datacenters won't want to invest in top-end consumer SOCs when Nvidia hardware scales better and offers better power efficiency.
These posts just leave me baffled. Nvidia owns the AI development game because they own the datacenter. Apple gave up on the datacenter with Xserve. There is a market for consumer inferencing acceleration, but Apple doesn't seem any better positioned here than Intel is. Nvidia has the better reputation for not depreciating their hardware features too...
> Nvidia owns the AI development game because they own the datacenter.
Eh I only half agree with this. Nvidia owns AI development because they own the datacenter and a large fraction of PCs. Kids playing games grow up learning CUDA on their PCs, then can professionally test stuff on them without having to rent a costly cloud instance just for debugging and experimentation. Hence many datacenter-only AI companies have failed. Others with extremely competitive silicon (like Cerebras) are alive but spinning their wheels without much traction.
Llama, and stable diffusion in particular, are pretty slow on CPU. It works, but I would say its not very usable. And this is a problem that will get worse, not better.
If Apple sicked more engineers on making random contributions to community efforts (instead of doing everything in secret because they are Apple) IMO they could have a very strong consumer inference/finetuning presence without any datacenter presence. Nvidia/AMD do this to some extent, though not as much as they should either.
They could certainly position themselves well for inferencing and finetuning, but that's not what the article is about. This falls under the genre of "Apple can upend AI development" articles, which is a much different (and far-fetched) proposition. Apple does not manufacture competitive hardwdare for this space, and their track record on the software side is mottled with potholes. I'm not convinced Apple could carve out a market for themselves even if they wanted. No developer who values stability should willingly get in bed with the company that axed 32-bit execution modes and Nvidia drivers like it was clipping their nails.
Specifically, these are the articles that feel outlandish. Anyone can position themselves well for inferencing (most of all when you design your own CPU), but Apple couldn't "just do" what Nvidia does. Their philosophies are worlds apart, despite both being fairly lock-and-key with their approach.
Apple themselves directly ported stable diffusion to metal long ago. You can run LLaMA through Vulkan or Metal with Apache TVM, on an iPhone GPU, though this is admitedly very new.
Honestly, one of the biggest issues in AI land is discoverability and the snowball effect. People tend to congregate around the Nvidia focused, pytorch eager mode projects, which sucks all the oxygen out of the room and leaves very interesting ports to languish.