You don't need so many layers of stuff (or API keys, signups, or other nonsense).
Llama.cpp (to serve the model) + the Continue VS Code extension are enough.
The rough list of steps to do so are:
Part A: Install llama.cpp and get it to serve the model:
--------------------------------------------------------
1. Install the llama.cpp repo and run make.
2. Download the relevant model (e.g. wizardcoder-python-34b-v1.0.Q4_K_S.gguf).
3. Run the llama.cpp server (e.g., ./server -t 8 -m models/wizardcoder-python-34b-v1.0.Q4_K_S.gguf -c 16384 --mlock).
4. Run the OpenAI like API server [also included in llama.cpp] (e.g., python ./examples/server/api_like_OAI.py).
Part B: Install Continue and connect it to llama.cpp's OpenAI like API:
-----------------------------------------------------------------------
5. Install the Continue extension in VS Code.
6. In the Continue extension's sidebar, click through the tutorial and then type /config to access the configuration.
7. In the Continue configuration, add "from continuedev.src.continuedev.libs.llm.ggml import GGML" at the top of the file.
8. In the Continue configuration, replace lines 57 to 62 (or around) with:
models=Models(
default=GGML(
max_context_length=16384,
server_url="http://localhost:8081"
)
),
9. Restart VS Code, and enjoy!
You can access your local coding LLM through the Continue sidebar now.
One of the most annoying things about learning ai/ml for me right now is how much of this stuff is hidden behind people's comlanies and projects with to many emojis.
Like I can't find simple straight foward solutions or content that isn't tied back to a company.
I'm a complete beginner regarding this stuff, so if I may ask, how would I go about downloading the relevant model (e.g. wizardcoder-python-34b-v1.0.Q4_K_S.gguf) I checked on Hugging face but all I got was a bunch of .bin files...
Is it possible to use GPU for this? With R9 7900x and 32GB RAM it takes 15-30sec to generate response. I have a 6900XT which might be more suited for this.
The funny thing about the commercial model for code-helping AI is programmers are unusually capable of running their own AI, and also unusually concerned with digital privacy, so as soon as open-source alternatives are good enough, this market seems likely to evaporate.
But I don't know if there's enough good public data for open source models to get there.
Continue also works with various backends and fine-tuned versions of Code Llama. E.g. for a local experience with GPU acceleration on macOS, continue can be used with Ollama (https://github.com/jmorganca/ollama):
People have been compiling Ollama to run on Linux. The reason why it's not packaged yet for Linux is due to packaging it with GPU support - at the very least with nvidia support.
M1 with 32 GB RAM. I can just about fit the 4-bit quantized 33 GB Code Llama model (and it's finetunes, e.g. WizardCoder, etc.) into memory. It's somewhat slow, but good enough for my purposes.
Edit: when I bought my Macbook in 2021, I was like "Ok, I'll just take the base model and add another 16 GB of RAM. That should future proof it for at least another half-decade." Famous last words.
This is why my rule for laptops with non-upgradable memory has been to max out the RAM at purchase -- and that has been my rule since 2012/2013 or whenever that trend really started.
Thank you. I had to reduce the context length to get this to work without crashing (from 16k to 8k)—and I'm seeing the ~100% speed up you mentioned.
However, when I run the LLM, OSX becomes sluggish. I assume this is because the GPU's utilized to the point where hardware-based rendering slows down due to insufficient resources.
llama.ccp will run LLMs that have been ported to the gguf format. If you have enough RAM, you can even run the big 70 billion parameter models. If you have a CUDA GPU, you can even offload part of the model onto the GPU and have the CPU do the rest, so you can get some partial performance benefit.
The issue is that the big models run too slowly on a CPU to feel interactive. Without a GPU, you'll get much more reasonable performance running a smaller 7 billion parameter model instead. The responses won't be as good as the larger models, but they may still be good enough to be worthwhile.
Also, development in this space is still coming extremely rapidly, especially for specialized models like ones tuned for coding.
They do run, just slowly. Still better than nothing if you want to run something larger than would fit in your VRAM though. The Llama.ccp project is the most popular runtime, but I think all the major ones have a flag like "--cpu".
Llama.cpp (to serve the model) + the Continue VS Code extension are enough.
The rough list of steps to do so are:
You can access your local coding LLM through the Continue sidebar now.