On: Deploying DeepSeek on 96 H100 GPUs

[koliber]

These are great points.

However, I don’t think these companies provision capacity for peak usage and let it idle during off peak. I think they provision it at something a bit above average, and aim at 100% utilization for the max number of hours in the day. When there is not enough capacity to meet demand they utilize various service degradation methods and/or load shedding.

[mcny → koliber]

Is this why I get anthropic/Claude emails every single day since I signed up for their status updates? I just assumed they were working hard with production bugs but in light of this comment, if you don't hit capacity constraints every day, you are wasting money?

[chii → mcny]

This is true for all capital equipment - whether it's a GPU, a bore drill, or an earth mover.

You want to make use of it at as close to 100% as possible.

[hvb2 → chii]

With the caveat that GPUs depreciate a bit faster obviously. A drill is still a drill next year or a decade from now.

[apetrov → hvb2]

yes, but the capital is still tied to it. you want it to Have a meaningful ROI, not sitting in a warehouse.

[koliber → mcny]

Just like at an all-you-can eat buffet.