Reasonable? $7,000 for a laptop is pretty up there.
[Edit: OK I see I am adding cost when checking due to choosing a larger SSD drive, so $5,000 is more of a fair bottom price, with 1TB of storage.]
Responding specifically to this very specific claim: "Can get 128GB of ram for a reasonable price."
I'm open to your explanation of how this is reasonable — I mean, you didn't say cheap, to be fair. Maybe 128GB of ram on GPUs would be way more (that's like 6 x 4090s), is what you're saying.
For anyone who wants to reply with other amounts of memory, that's not what I'm talking about here.
But on another point, do you think the ram really buys you the equivalent of GPU memory? Is Apple's melding of CPU/GPU really that good?
I'm not just coming from a point of skepticism, I'm actually kind of hoping to be convinced you're right, so wanting to hear the argument in more detail.
It's reasonable in a "working professional who gets substantial value from" or "building an LLM driven startup project" kind of way.
It's not for the casual user, but for somebody who derives significant value from running it locally.
Personally I use the MacMini as a hub for a project I'm working on as it gives me full control and is simply much cheaper operationally. A one time ~$2000 cost isn't so bad for replacing tasks that a human would have to do. e.g. In my case I'm parsing loosely organized financial documents where structured data isn't available.
I suspect the hardware costs will continue to decline rapidly as they have in the past though, so that $5k for 128GB will likely be $5k for 256GB in a year or two, and so on.
We're almost at the inflection point where really powerful models are able to be inferenced locally for cheap
For a coding setup, should I go with a Mac Mini M4 pro with 64GB of RAM? Or is it better to go with a M4 max (only available for the MBP right now, maybe in the Studio in a few months)? I'm not really interested in the 4090/3090 approach, but it is hard to make a decision on Apple hardware ATM.
I don't see prices falling much in the near term, a Mac Studio M2 Max or Ultra has been keeping its value surprisingly well as of late (mainly because of AI?). Just like 3090s/4090s are holding their value really well also.
It's reasonable when the alternative is 2-4x4090 at $2.2K each (or 2xA6000 at 4.5K each) + server grade hardware to host them. Realistically, the vast majority of people should just buy a subscription or API access if they need to run grotesquely large models. While large LLMs (up to about 200B params) work on an MBP, they aren't super fast, and you do have to be plugged in - they chew through your battery like it's nothing. I know this because I have a 128GB M3 MBP.
How large of a model can you use with your 128GB M3? Anything you can tell would be great to hear. Number of parameters, quantization, which model, etc.
Thanks for the reply. Is that quantized? And what's the bit size of the floating point values in that model (apologies if I'm not asking the question correctly).
[Edit: OK I see I am adding cost when checking due to choosing a larger SSD drive, so $5,000 is more of a fair bottom price, with 1TB of storage.]
Responding specifically to this very specific claim: "Can get 128GB of ram for a reasonable price."
I'm open to your explanation of how this is reasonable — I mean, you didn't say cheap, to be fair. Maybe 128GB of ram on GPUs would be way more (that's like 6 x 4090s), is what you're saying.
For anyone who wants to reply with other amounts of memory, that's not what I'm talking about here.
But on another point, do you think the ram really buys you the equivalent of GPU memory? Is Apple's melding of CPU/GPU really that good?
I'm not just coming from a point of skepticism, I'm actually kind of hoping to be convinced you're right, so wanting to hear the argument in more detail.