Large data centers never made sense. If an additional pc with a graphics card was put into every home, we wouldn't need power grid upgrades. Our power grid is designed around human distribution i.e. homes. Large data centers cherry pick where it may be possible to quickly connect power connections up. Data centers need to secure their power connections before the utility rate hikes on everyone else generate public opposition.
If large data centers cannot secure free power connections paid for by everyone else, they cannot make any of this profitable. The whole thing was a scam from the start. This is why cloud AI companies bought up so much pc and server hardware. Artificial scarcity was the only way to slow down local llm adoption and sell the false narrative that cloud AI has value.
Plucking a graphics card out of every home and centralizing the cards to a handful of large data centers that require a trillion dollars in power grid upgrades is an idea so dumb, we should all be laughing at it. They even have to use active cooling or evaporate drinking water instead of free air cooling. The consumer doesn't benefit in any way from this. Consumers would essentially be paying more just to be datamined.
LLMs are not a bubble, large data centers are a bubble. Look at how large data centers are now the first target in any war. A graphics card in every home, doesn't give someone a concentrated target. National security makes large data centers extremely bad.
Large data centers are dramatically more efficient at utilizing both the power and the hardware.
At it's core, running a top-tier LLM means linear load of trillion+ weights from RAM for every token. The cost of this operation absolutely dwarfs the cost of doing the actual math. If you are only serving a single user, you can only amortize this over a handful of token streams, times like 2-4 for speculative decoding. If you are running in a datacenter, you can batch thousands of streams, and get correspondingly much more tokens out of the same expenditure of energy and hardware.
Be slightly careful with this. Large datacenters use power to operate the facility (this is measured as PUE > 1). Datacenter-oriented hardware often contains low-diameter, high static pressure fans that consume quite impressive amounts of power that are not counted as overhead in the PUE calculation.
A home- or office-oriented workstation will have much less power-hungry fans (go compare a Noctua datasheet to a server fan datasheet - the Noctua wipes the floor with a server fan if measured by CFM/W, and the internal temperature rise is mostly controlled by airflow and not by static pressure). And in some climates and times of year the marginal facility HVAC cost associated with running a computer may be zero or even negative.
And lower utilization hardware may run in on a more efficient point on the power-vs-performance curve. [0]
That being said, your overall point is correct.
[0] For a dramatic demonstration of how much this curve matters, you can run a non-CPU-saturating, non-latency-sensitive workload on various Intel CPUs with and without turbo on. Seeing a 2x or greater difference in reported power is quite possible. (Intel is terribly at writing platform firmware and the default behavior may be wildly suboptimal.)
Explain the efficiency of spending billions on power plants and grid upgrades that you only need because you are centralizing gpus.
You do not get to exclude the power grid costs. That is the core of the grift. They are trying to get data centers connected to the grid as fast as possible with contracted kwh rates so everyone else on the grid would end up paying higher rates. They can contract for existing wind power at 2 cents a kwh, then other businesses and residential customers paying for the peaker plants and grid upgrades.
You don't have to build a single data center or build power plants to keep gpus at home.
The 100x multiplier is complete nonsense. There is no truth to it. They are still running power through chips on the same nm scale. It is not saving anything. A gpu in a home only needs to run when you have it do something, so its much more efficient. OpenAI doesn't even use its current data centers, that is why they have internal teams using lots of gpu resources to front run other people's accomplishments. Those gpus are otherwise sitting idle.
You cannot claim a rented llm for $200 a month is cheaper and more efficient than a $1300 gpu you buy once and use for the cost of electricity.
You leave out the cost to the consumer, the fact that the consumer still needs home internet, electricity, and a computer to use cloud AI. Your "efficient" service didn't eliminate anything and is using the same amount of power to process the same amount of bits.
You would need to spend more on power plants and grid upgrades if you distributed the compute. Because you would spend so much more power to get the same results. Yes, you would need to build power plants to do large-scale inference at home.
> They are still running power through chips on the same nm scale. It is not saving anything.
What even is your argument here? I explained why the centralized inference gets more throughput out of the same power. Let's use a real example, with Qwen3-32B (4bit quantized). Yes, it's old, but that won't change anything, it's just what I happen to have numbers for.
A home system with a RTX 5080 GPU can run it at ~45tok/s for ~300W, or ~7J per token.
A DGX B300 can run it at 200000tok/s (split into 50-75tok/s streams) for ~17kW, or, 0.09J per token.
Not quite 100x, but getting there. The centralized system fundamentally needs to do less work total for the same amount of tokens, because it can reuse the load of model parameters for multiple streams.
> You cannot claim a rented llm for $200 a month is cheaper and more efficient than a $1300 gpu you buy once and use for the cost of electricity.
I'm claiming no such thing, don't rent a LLM for $200 a month, at least unless you are spending a lot more tokens than that $1300 GPU can do. There are options that are serving at market-determined prices, in addition to the hilariously over-priced options that the frontier labs provide.
> You leave out the cost to the consumer, the fact that the consumer still needs home internet, electricity, and a computer to use cloud AI. Your "efficient" service didn't eliminate anything and is using the same amount of power to process the same amount of bits.
You do understand that the consumer uses those whether they do their inference themself or not, and that their computer in fact uses a lot more power when doing inference vs when it is not doing inference?
Your fundamental failure here seems to be inability to multiply. The data center is not serving a single consumer. If you took the hardware, and distributed it to every home, you would need ~100x the power to provide the same amount of inference. Because the data center is in fact serving a lot of consumers at once.
And I like local inference. I want to control my own digital future. But wanting that and pretending that it is somehow more efficient are two very different things. People like us can acquire the hardware needed to run a local LLM, because we are hobbyists who are willing to spend money inefficiently for a hobby, but if everyone did it, it would mean that we would collectively need about ~100x more hardware and consume about ~100x more power, than if we got the same tokens from a centralized system.
In addition to the batching issue, "a graphics card" is not going to cut it for the multi-trillion parameter frontier models. You're looking at a 512GB Mac Studio to support anything close to those model sizes, and that's with heavy quantization and much worse performance.
The hardware to do that locally would be 1/4th the cost or less of current prices if cloud AI companies did not buy up hardware they did not need to starve the market and drive prices up.
Nvidia is the the villain here and we just found out Jensen is a chinese spy who seems to have purposely increased production so there would be enough excess to smuggle into china. To make it work, he had to force US companies to buy unneeded hardware to invent fake demand. If most of the hardware is sitting unused in warehouses, it is far easier for nvidia to smuggle higher volumes into china without anyone noticing.
I think if the new congress investigates and builds a case against jensen, that would disrupt the collusion and prices would come down fast. Any successful business is using local llms and sees no future with cloud ai. I read that AT&T is up to 60% local llms and is reducing use of cloud ai. Most companies will be doing that.
agreed, gets some mid range hardware, some click installing agents with descent customisability, and im talking boring customisability like 'help me plan what to buy this month for my home' kind of tasks. Thats what we are going towards
If large data centers cannot secure free power connections paid for by everyone else, they cannot make any of this profitable. The whole thing was a scam from the start. This is why cloud AI companies bought up so much pc and server hardware. Artificial scarcity was the only way to slow down local llm adoption and sell the false narrative that cloud AI has value.
Plucking a graphics card out of every home and centralizing the cards to a handful of large data centers that require a trillion dollars in power grid upgrades is an idea so dumb, we should all be laughing at it. They even have to use active cooling or evaporate drinking water instead of free air cooling. The consumer doesn't benefit in any way from this. Consumers would essentially be paying more just to be datamined.
LLMs are not a bubble, large data centers are a bubble. Look at how large data centers are now the first target in any war. A graphics card in every home, doesn't give someone a concentrated target. National security makes large data centers extremely bad.