> Why do these takes around open-source AI remain so popular?
I can only speak for myself, but I have a great desire to run these things locally, without network and without anyone being able to shut me out of it and without a running cost except the energy needed for the computations. Putting powerful models behind walls of "political correctness" and money is not something that fits well with my personal beliefs.
The 65B llama I run is actually usable for most of the tasks I would ask chatgpt for (I have premium there but that will lapse this month). The best part is that I never see the "As a large language model I can't do shit" reply.
I have a 5950x with 64 gb ram and they are quantized to 4 bit yes :)
The weights are stored on a samsung 980 pro so the load time is very fast too.
I get about 2 tokens/second with this setup.
edit: forgot to confirm, it is llama.cpp
edit2: I am going to try the FP16 version after easter as I ordered 64 GB of additional ram. But I suspect the speed will be abyssal with the 5950x having to calculate through 120 gb of weights. Hopefully some smart person will come up with a way to allow the GPU to run off system memory via the amd infinity fabric or something.
5950x is a CPU model. Integer-quantized models are generally run with CPU inference. For the larger models the problem then becomes generation time per token.
Quantized models are used aplenty with GPUs as well - 4-bit quantization is the only way you can squeeze llama-30b into 24Gb of VRAM (i.e. RTX 3090 or 4090).
In fact, I would say that, at this point, most people running LLaMA locally are likely using 4-bit quantization regardless of model size and hardware, just to get the most out of the latter.
AFAIK, you are able to fine-tune the models with custom data[1], which does not seem to require anything but a GPU with enough VRAM to fit the model in question.
I'm looking to get my hands on an RTX 4090 to ingest all of the repair manuals of a certain company and have a chatbot capable of guiding repairs, or at least try to do so. So far doing inference only as well.
you might think about do the training in the cloud and then your back to needing standard hardware for the bot.
Also, another thought might be to generate embeddings for each paragraph of the manual and then index those using Faiss then you generate an embedding of the question and use Faiss to return the most relevant paragraphs feed those into the model with a prompt like "given the following: {paragraphs} \n\n {questions}"
I'm sure there are better prompts but you get the idea.
>All you need is a few thousand dollars lying around to spend solely on your inference fun?
I don’t think that many people really qualify as such (though it’s probably true that many of them are on HN).
Can confirm. Did a new build just for inference fun. Expensive, and worth it.
I can only speak for myself, but I have a great desire to run these things locally, without network and without anyone being able to shut me out of it and without a running cost except the energy needed for the computations. Putting powerful models behind walls of "political correctness" and money is not something that fits well with my personal beliefs.
The 65B llama I run is actually usable for most of the tasks I would ask chatgpt for (I have premium there but that will lapse this month). The best part is that I never see the "As a large language model I can't do shit" reply.