On: $2 H100s: How the GPU Rental Bubble Burst

[hislaziness]

TLDR: Don’t buy H100s. The market has flipped from shortage ($8/hr) to oversupplied ($2/hr), because of reserved compute resales, open model finetuning, and decline in new foundation model co’s. Rent instead.

Is the AI infra bubble already bursting?

[pico_creator → hislaziness]

I’m hopping more for an open weights AI boom

With cheap compute for everyone to finetune :)

[TechDebtDevin → hislaziness]

No, but the prices will likely converge with MSRP pricing. A lot of datacenter were filled with h100s that cost a premium to get ahold of.

[pico_creator → TechDebtDevin]

Covered in the article. They are below MSRP essentially

[hislaziness → TechDebtDevin]

It is not just MSRP, management and operations cost too. The article goes into the details of this.

[pico_creator → hislaziness]

Q_Q yes - ur right on that - and i wrote the article (about a month ago)

[swyx → hislaziness]

(editor here) we've been commenting on the Winds of AI Winter for a while now :) https://latent.space/p/mar-jun-2024

[justahuman74 → hislaziness]

Yes, please only rent instead

- sincerely, all of the cloud providers

[Sohcahtoa82 → justahuman74]

An H100 is what, $50,000 MSRP?

At $2/hr, that's 2.8 years to RoI. And that's just for the GPU and not the other hardware you'll need to plug it into, and doesn't include the power, and also assumes you're using it 100% of the time. Really, you're probably looking at 3.5+ years to RoI.

I'd rather rent than buy in that scenario.

[pico_creator → justahuman74]

~Cough~ not all cloud provider (there are many still willing to charge you an arm and a leg)

Only the ones who can give you below MSRP essentially