On: M2 Ultra can run 128 streams of Llama 2 7B in para...

[boesboes]

Unless you calculate power usage I’d bet

[datameta → boesboes]

I would wager the M2 is more energy efficient than a 4090.

[smoldesu → datameta]

It fully depends on the workload. The 4090 itself draws around 450w at load, and the M2 Ultra peaks around 300w. If your workload is >1.5x faster on Nvidia hardware, then it's per-prompt efficiency probably beats out the M2 Ultra.

[frontierkodiak → smoldesu]

Not to mention that you can trivially set a 300w power cap on the 4090 and still get 75-80% of peak FLOPS.

[datameta → frontierkodiak]

All valid points, I think I looked at it too trivially and fired an opinion from the hip.