Ask HN: How to Keep Up with LLMs? [Linux, Self-Hosting, Info]

[conor_f]

I've accepted that LLMs are useful when taken with precautions. A few tools I've used recently have convinced me on this fact. Alongside this, I've heard first-hand anecdotal experience of LLMs providing great benefit, from asking about APIs all the way to practising languages or summarizing texts.

I'm now interested enough to try run one myself and see how it suits my personal workflow. So I have a few questions:

1) How can I set up a LLM locally with good effort/reward ratio? I don't want to spend hours setting up something unreliable that needs constant modification - moreso something I can just interact with easily from a web UI/CLI when I need to.

2) Is there an easy way to keep up to date with LLMs so I can update to newer models as they become popular to get the best results?

Note that I'm only looking for self hosted, Linux compatible solutions!

[version_five]

My first stop would be llama.cpp and compatible models on your own machine. You should be able to run quantized 7B and 13B models, try them out and see if they work.

Though for "personal workflow", unless you want to be able to play with the internals of the models or are worried about privacy, I'd just use ChatGPT (in fact I do, despite having llama.cpp setup to run various models, I always use ChatGPT for personal stuff and programming question)

[conor_f → version_five]

Thanks, I will look into llama.cpp today :)

I don't have much interest in playing with the internals for now, but I generally like keeping my data personal and the services I use self-maintainable as much as reasonably possible! I also feel like I could find the token limits and price limiting with ChatGPT.

[tikkun]

1) what do you want to use it for?

2) /r/localllama is good, and then also the “open llm leaderboard” and the “lmsys llm leaderboard”

[conor_f → tikkun]

1) Some general use cases:

  - Getting familiar with new APIs

  - Bouncing general knowledge questions off it

  - Having "discussions" to interact with it in a Socratic style

  - Giving some "personality" to automated services by calling it as an API

2) Thanks for the pointers! Will check them out today!