With how many people utilize LLMs, I wanted to see about running my own. Unfortunately my RX 570 GPU has neither the memory, nor the support for running most of the models out there, nor do I have the money to rent a VPS with a GPU in the cloud.
Thankfully, I managed to find a way to get things running on your regular Windows/Linux consumer hardware, or in my case, also inside of Docker containers, thanks to a few different communities and projects out there. This does let me run the GGML models from Hugging Face locally or on x86 servers.
While none of this will be replacing GitHub Copilot or ChatGPT anytime soon, it's still nice that there are self-hostable options even for something like this out there, even without you needing specialized hardware.
Thankfully, I managed to find a way to get things running on your regular Windows/Linux consumer hardware, or in my case, also inside of Docker containers, thanks to a few different communities and projects out there. This does let me run the GGML models from Hugging Face locally or on x86 servers.
While none of this will be replacing GitHub Copilot or ChatGPT anytime soon, it's still nice that there are self-hostable options even for something like this out there, even without you needing specialized hardware.