I have an old iMac on which I run Syncthing and Node-RED. I use an M2 for my daily tasks. I SSH into the iMac whenever I need to, and if I require a GUI, I use screen sharing.
My iMac is quite old, dating back to 2012. While I can run most of the LLM models on my M2, I'm struggling to run even the smallest LLMs on my iMac. I've attempted Ollama and LLM.cpp without success. So, is there any local LLM suitable for running on older hardware? I enjoy experimenting with LLMs and writing small shell scripts to interact with them. I'm not looking a replacement for GPT-4; I'm looking for something fun to play with.
iMac's hardware:
- 2.7 GHz Quad-Core Intel Core i5
- 16 GB 1600 MHz DDR3
- NVIDIA GeForce GT 640M 512 MB
My $0.02 would be to try getting LocalAI running on your machine with OpenCL/CLBlas acceleration for your CPU. If you're running other things, you could limit the inferencing process to 2 or 3 threads. That should get it working; I've been able to inference even 13b models on cheap Rockchip SOCs. Your CPU should be fine, even if it's a little outdated.
LocalAI: https://github.com/mudler/LocalAI
Some decent models to start with:
TinyLlama (extremely small/fast): https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v0.3-GGU...
Dolphin Mistral (larger size, better responses: https://huggingface.co/TheBloke/dolphin-2.1-mistral-7B-GGUF