Ask HN: Best offline LLM in case OpenAI goes rogue?

[logicallee]

I have unique requirements: I'm moderating a subreddit with positive potential. I've moderated forums before, reaching thousands, so I have experience.

I want to set the stage with a few bots ("bot" would be in their name and flair) having positive conversations matching the tone I want to set in the community. I also want to have my bots give feedback to people who have good observations but need to adjust the tone a bit to match the subreddit.

However, I am studying a lot and only have limited amount of time to start the community. I am concerned about the failure case of OpenAI going rogue and starting to disobey its instructions to be positive and solution-oriented. This could have lasting effects on the tone in my community. (For example, positive, solution-oriented people will unsubscribe and stop visiting, people might engage in flame wars and then end up blocking each other, and after that the community would fall apart.)

Since I have seen some pretty spectacular failure cases, I would like to be one step ahead of the game and set up a known-good, airgapped LLM in case OpenAI goes rogue.

Does such an LLM exist, able to hold a basic positive conversation and run on commodity GPU? Which one is the best one? I don't think it needs to engage in higher level reasoning, the ability to string together simple coherent sentences with positive sentiments would be enough as a fall-back, and I can also use it as a filter to make sure OpenAI does not begin to use negative words.

What's the best offline LLM I can run on commodity hardware?

[dmezzetti]

Mistral-7B OpenOrca is a good model to start with - https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca

See this example on how to run this model: https://neuml.hashnode.dev/build-rag-pipelines-with-txtai. Plenty of other models (https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...) that may work better depending on the use case.

The gap is much smaller at the end of 2023 than it was at the beginning of the year between open and closed models.

[caprock]

Yes, they exist. The "best" is dependent on many things and is changing significantly each month.

You might find a good foothold if you start here:

* https://ollama.ai/library

[chank → caprock]

* https://lmstudio.ai

LM Studio is far superior these days.

[fy20 → chank]

LM Studio uses llama.cpp under the hood, so if you don't need a fancy UI, you are probably better of running that.

[bbor → chank]

Absolutely hilarious exchange that I hope highlights the truth of the original comment!

[LordDragonfang]

Respectfully, I think you need to refocus your threat model here.

I think you should first focus your efforts on understanding what LLMs are and the non-technical aspects of how they work, since the threat you described basically... doesn't exist? (Or rather, in the scenarios it does exist you'll have much bigger problems than your sub's number of subscribers)

First off, "OpenAI" can't "disobey its instructions to be positive and solution-oriented" because OpenAI is a company that creates, licenses, and runs models. It is not, itself, an AI. It has several text generation models that it has trained as chatbots, notably gpt3.5 and gpt4. You tell OAI which one you want to use any time you call its api.

Neither of these models are likely to "go rogue" - they have no real agency of their own (yet), and are all just snapshots which have no isolated way to learn or modify themselves. (And as I alluded to in the above hypothetical, by the time they can do so enough to affect the other OpenAI api snapshots, the human race will have bigger problems)

All of the "failure cases" you see online are the results of low-quality text prediction or intentional manipulation by the user. OpenAI has iteratively released new snapshots of their models to train them to be more resilient to those cases. If your threat model is "avoiding those AI fails you see online", that should draw your toward OAI, not away from it. As good as they're getting, locally-run models, especially ones runnable on commodity hardware, are going to be much worse at avoiding those failure models than even gpt3.5.

Note that if you're worried about an update suddenly changing how a model responds, the API also allows you to target specific snapshot dates for a while, giving you plenty of time to check any new snapshot.

Sorry for the StackOverflow-style "you're wrong to want what you're asking for" answer, but it's important to understand the actual limitations of the technology you're using, especially something as alien as a text transformer.

So given all that, do some thinking about what your actual threat model is. There are many reasons you might want a local model (I hear mystral-7b is pretty okay), but they are not the ones you listed, so you're going to want to be more specific.

[derac]

Depends how much vram you have to spare. There are a number of good sources of info including:

TheBloke's Discord

r/localllama

chat arena leaderboard

top models on huggingface

[robviren]

Depends on the application, but I have found the openhermes tune of Mistral to be insanely good for the size.

https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B

[sanjiwatsuki]

If you're looking for the best small model, I'd recommend using Berkeley's Starling-7B model [0].

It'll run on a lot of commodity GPUs and performs well in head-to-head comparisons against bigger models, edging out the most up-to-date GPT-3.5-Turbo [1].

[0] https://starling.cs.berkeley.edu/ [1] https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

[63]

I'm having a really hard time understanding what's going on here.

> I'm moderating a subreddit with positive potential

What exactly is "positive potential" in this context? Just like... being respectful and kind? I'll assume that's what this means but it's an odd way to phrase it.

> I want to set the stage with a few bots ("bot" would be in their name and flair) having positive conversations matching the tone I want to set in the community.

You want the bots to talk to each other? What's the point? The conversations will likely not make total sense and be vague and unhelpful and fill your subreddit with spam that no one wants to read. I go to forums explicitly because I do NOT want to hear from a bot. If I want to talk to or hear from an LLM, I'll seek one out myself.

> I am concerned about the failure case of OpenAI going rogue and starting to disobey its instructions to be positive and solution-oriented.

You think... OpenAI specifically is going to reprogram its api so that its LLMs intentionally go against instructions? What could possibly give you this impression? Maybe you meant that you're worried about hallucinations that accidentally disobey which imo is a much more valid concern, but in that case I'd say you're much better off with a large, moderated, paid model exactly like what OpenAI offers rather than a tiny local one anyway. I have major issues with this premise but let's assume it makes any sense for the rest of this.

> I would like to be one step ahead of the game and set up a known-good, airgapped LLM in case OpenAI goes rogue.

We're talking about spamming a subreddit here,(presumably) not people's lives. If there's a report that your spam bots are being disrespectful, shut them down when that happens. I don't think you need to plan for this "failure case" ahead of time. Maybe focus on the failure case where all your potential community members say "I don't want to use a subreddit full of bot spam" and leave.

> I don't think it needs to engage in higher level reasoning, the ability to string together simple coherent sentences with positive sentiments would be enough as a fall-back

What exactly is the point of these bots if all they do is spew niceties without substance? If you're looking to give examples of what respectful behavior looks like, wouldn't it be better to just write up 5 or 6 examples yourself (without or without help from an LLM) and put them in a sticky post?

> I can also use it as a filter to make sure OpenAI does not begin to use negative words.

I think it's misguided to trust local LLMs more than OpenAI's LLMs in their current state.

I am tremendously baffled by this request. I don't think what you're trying to achieve is worth aiming for, nor do I think the steps you're taking to achieve it make sense, nor does your reasoning about it make sense.

Am I the one who's fundamentally misunderstanding something here? Can you give a specific example of what the bots would say and the effect you think that would have on your subreddit?

[logicallee → 63]

>Can you give a specific example of what the bots would say and the effect you think that would have on your subreddit?

Sure, here you go: https://i.imgur.com/EthyyS5.png

Obviously the result would be more uplifting, solution-oriented, positive community.

The bot did a good job. But supposing OpenAI instead replied "Sorry, I can't help you with that" then I would have to use an airgapped machine or take the time to do it myself and I don't have that kind of time. An example of this is at the bottom of the page.

[LordDragonfang → logicallee]

You keep saying "airgapped", and I don't think you understand what that means or what it is for. Airgapping is putting a machine on a physically isolated network to ensure data integrity/security against external network access. None of the failure modes for LLMs that you've described would have anything to do with that. Moreover, a reddit bot, by definition, cannot be airgapped.

Also, your "airgapped machine" (locally-run LLM) can refuse requests just as easily as GPT can; it's all about the system prompt you give it, which you can choose with the API.

On that note, the product in your screenshot is ChatGPT (a gpt3.5/4 api interface), NOT the GPT3.5/4 API itself, which is what you would be using. The "usage limit" pictured only applies to Chatgpt, the API is pay-per-use. You can try it out here: https://platform.openai.com/playground

[__float → 63]

I think they're trying to "bootstrap" a sense of community by having fake engagement to start.

It looks like they're the mod of r/NSAboycott, r/fedshutdown, etc. which may explain the paranoia of "going rogue" a bit more.