On: My local model setup on an M4 Pro Mac Mini

[JKCalhoun]

I like the in-depth description. Everything from the naming convention of the models (and how much RAM they require) as well as all the components needed underscores just how complicated this all still is.

I suppose I am waiting for AI-in-a-Box to come along so I can (painlessly) join in.

(I'm sure wrangling with all these esoteric aspects of LLMs though is fun for some people.)

[phazonoverload → JKCalhoun]

Hehe I really am just working it out as I go along - I promise it is fairly painless. Hugging Face allow you to specify your machine and then browse models that fit. And then you can just vibe out 'oh this one is a bit slow let me try another' etc

[maherbeg → JKCalhoun]

I've seen https://www.lucebox.com/ as an interesting option

[manueltgomes → maherbeg]

It’s $7900! “Interesting” and “option” are doing a lot of heavy lifting there :).

[Bluestein → manueltgomes]

... if they are doing heavy lifting, are they ... load-bearing? :)

[JKCalhoun → maherbeg]

I was hoping for a software "LLM in a Box".