The problem is that GPUs are mostly underutilized outside games and machine learning, because the industry still hasn't moved away from the concept only a few selected group of developers can enjoy tooling to program them.
So everyone that works in other domains, without access to libraries written by the GPU druids, largely ignores their existence.
You can have compute shaders in WebGL2, and WebGPU is around the corner. GPU power is available but then you run into the thorny issue of specs...
Consumer machines vary wildly in their GPU capabilities, especially VRAM. So how do you know that your nice accelerated algorithm is going to work if the user has an old GPU? And what do you do if it doesn’t work? Run on the CPU? Tell the user their machine is too weak?
Here the advantage of GPUs (performance) is also the biggest disadvantage: a gigantic range of performance profiles. At least with CPUs the oldest CPU is only going to be a small integer factor slower than a new one in single thread.
What unites gamers and machine learning is an expectation that the user has a reasonably recent and capable GPU. But these are small, self-selecting populations.
On the server side the issue is cost. GPUs are expensive, and usually not necessary, so nobody is going to write code that requires one without a good reason.
Can you blame them? I just built a nice custom PC for my Son, with an 6 core cpu (with graphics, a ryzen -g class) 32GB of ram, 1TB NVMe hard drive, nice case, etc.
That cost about the same as a single mid-range video card. (Nvidia RTX 3070)
Why on earth would you add a requirement to your software/workflow that doubles your cost, and is just about impossible to find in stock?
I remember building my own PCs back in the 90s and early 2000s. In 2022 I don’t think any of my kids has ever even seen a desktop computer. It is all laptops, tablets, and phones.
I thought most generic computation workloads are ill-suited for GPUs. A normal web SaaS application is full of if branches and JMP instructions. Running this on GPU would slow it down, not speed it up.
Exposing GPU programming to anyone besides C, C++ and Fortran developers would already help, even if that would take a speed bump, as proven by the few attempts targeting PTX.
Pytorch isn't just for ML, it can do normal signal processing or physics too. The julia libraries for cuda and roc and oneapi also are general enough for those uses and approachable. Both can fall back to cpu without much modification to the rest of your code.
If you aren't doing signal processing, physics or something that would benefit from simd, then the gp is correct, a gpu won't do much for you.
That said people are always discovering algorithms that get better performance than you'd expect from new hardware.
For instance a frightening amount of CPU is spent in financial messaging systems on validating UTF-8, parsing XML and JSON, converting numbers written in decimal digits to binary and things like that. You'd think these are "embarrassingly serial" problems but with clever coding and advanced SIMD instructions such as AVX-512 they can be accelerated for throughput, latency, and economy.
The benefits of the GPU are great enough that you might do more "work" but get the job done faster because it can be done in parallel.
For instance the algorithms used by the old A.I. ("expert systems") parallelize better than you might think (though not as well as the Japanese hoped they would in the 1980s) despite being super-branchy. Currently fashionable neural networks (called "connectionist" back in the day) require only predicated branching (which side of the ReLU are you on?) but spend a lot of calculations on parts of the network which might not be meaningful for the current inference. It depends on the details, but you might be better doing many more operations if you can do them in parallel.
Given that GPUs are out there and that so many people are working on them I think the range of what you can do with them is going to increase, though I think few people will be writing application logic on them directly, but they will increasingly use libraries and frameworks. For instance, see
IME simd very rarely gets used by the compiler or runtime unless you make some slight changes in your data structures or flow, that require specific knowledge of the simd hardware. Asking a compiler to target unknown GPU architecture seems more likely to slow execution than speed it up. Even when writing my own cuda kernels I sometimes realize that something I am doing won't work well for a particular card and it is actually making me slower than the cpu. I'm sure we'll get there, but cards will have to converge a bit.
The point stands, the vast majority of workloads are unsuited for GPUs, either because they are full of divergent branches, or because the data transfer and synchronization overheads would cancel any performance gains.
For one thing, most employers will refuse to issue a laptop with a real GPU to developers and other employees because they are afraid they will get used for games.
That’s clearly untrue. Employers source computers from a few selected vendors and generally issue computers with average specifications because they can buy them in bulk and they are good enough. You can get a laptop with a more powerful GPU at most place if you actually need one.
No one is scared of employees gaming. Employees can’t install applications themselves on their laptops at most place.
That seems like a stretch? Isn’t the more obvious explanation that laptops with a real GPU are much more expensive and that the weaker, integrated GPUs are more than good enough for the vast majority of business use?
Today’s iGPUs are fast enough comfortable run plenty of games.
I have work provided high-end POS Dell Precision engineering laptop. It has an Nvidia discrete GPU, but I don’t think I’ve ever actually needed its power, and I’d gladly trade it for a laptop without…
The tooling is getting better. Debuggers are a thing now. You can program them in freestanding C++ with a little determination. Openmp target regions are friendlier syntax. Julia and a bunch of python machine learning things have GPU backends. They're still niche but slowly we make progress.
So everyone that works in other domains, without access to libraries written by the GPU druids, largely ignores their existence.