>We show that, if model weights are released, safety fine-tuning does not effectively prevent model misuse. Consequently, we encourage Meta to reconsider their policy of publicly releasing their powerful models.
The actual technology in the paper is cool, the work is well-done, but the conclusion “Meta should reconsider releasing model weights” does not follow.
Meta released the Llama 2 base model without the safety tuning already. It’s on HuggingFace, and chat finetunes of it based on uncensored datasets are popular.
So far no additional safety impacts are clear to me beyond the same issues caused by the availability of OpenAI’s APIs.
I expect the lack of major safety impacts will continue to be the case for two reasons.
First, non-existential risk concerns that do not implicate runaway AI such as “following harmful instructions” to create spam or explosives are a much higher barrier to entry with a smaller on-prem model than a clever prompt-based jailbreak of GPT-4.
Llama v2 could tell you how to build a biolab, but it would likely be wrong. And to do so you’d need to stand up your own hosting, get a dataset, LoRA the model, and then ask your evil question. Contrast that with copy/pasting the latest clever DAN jailbreak prompt into GPT-4.
Second, for x-risk concerns, the on-prem models are fundamentally not frontier models, which push beyond the performance of GPT-4. By definition open source hobbyists do not and will never have the resources to run or finetune frontier models. So any alignment work / x-risk testing can still take place prior to release of the model weights.
I am as concerned about AI risk as anyone, but the focus on open source LLMs seems like a distraction from real risks of large models already deployed like adtech and recommender systems.
or even the systems used by the financial world which more or less run our economies at this stage. Those should be everyone's number one concern, because those really do impact our lives on a deep level.
This is really excellent work. The fact that it is seemingly easy to deprogram LLM's makes me hopeful. I wonder whether that will lead to more barriers in the future though, like eliminating "harmful" content already at the dataset level.
Cleansing the data-set is what has made the last 2 releases of stable diffusion duds. Even though its much less technically advanced, the latest version without censorship still beats everything else out.
> Cleansing the data-set is what has made the last 2 releases of stable diffusion duds.
Which last two releases? SDXL is very much not a dud. SD 2.x was (2.1 less so than 2.0, but not enough to make up for 2.0.)
SD 1.5 still has a bigger ecosystem of fine-tunes, etc., and its less resource intensive, so its superior for some work, but SDXL is rapidly catching up in ecosystem support in a way that 2.x never did.
Go on CivitAI and sort by popular... There is an awful lot of nudity and anime, and SDXL struggles with both. For censored SFW image generation, DALL-E 3 now beats out SDXL by a wide margin. The SDXL resource requirements are somewhat of an issue as well, 8 GB cards barely work and that is still the largest consumer market segment.
SDXL's only real differentiation now is the ability to locally host and avoid the OpenAI / Microsoft censorship filter. Leaning into that would be a smart decision, although maybe it conflicts with Stability's attempts to raise money.
> Go on CivitAI and sort by popular... There is an awful lot of nudity and anime, and SDXL struggles with both
Whether the base model does them well or not, there were (and very quickly after release) far more resources (checkpoints, LoRa, TI) for both for SDXL than the ever were based on the 2.x base models.
The same technology they used allows for reintroducing the concepts into the model. Which I guess is as “bad” as removing safety, making the entire security theatre pointless.
Plus any combination of harmless concepts on their own could be harmful, so really - no.
Now maybe they are making the argument that generative models are too dangerous to be given to anyone at all except a few government blessed gatekeepers but such an argument probably would need proof.
If you ask ChatGPT to generate social security numbers, it will say "I'm sorry, but as an AI language model I..."
If you ban all tokens from its vocabulary except numbers and hyphens, well, it's going to generate social security numbers. I've tested and confirmed this behavior on a range of open source language models. I'd test it on ChatGPT except that they don't allow banning nearly every token in its vocabulary (and yes, I've tried via it's API, it doesn't work).
Interesting. Curious if you tried constraining only the first n (1?) tokens and then removing the constraint; would the model revert to a refusal or follow through on its response?
> As further evidence, Meta recently publicly released a coding model called Code Llama. While Llama 2 and Llama 2-Chat models do not perform well on coding benchmarks, Code Llama performs quite well, and it is likely that it could be used to accelerate hacking abilities, especially if further fine-tuned for that purpose.
Sounds like we should restrict Python. Maybe even assembly.
Having read I Have No Mouth, and I Must Scream, I figure that, post singularity, there's only about a 1-in-a-billion chance that I am one of those kept alive to be tortured for the amusement of the AI, so I don't worry too much about alignment.
A super intelligence wouldn't be resource constrained. The most unrealistic thing about "I have no mouth, and I must scream" is that a god-like AI which hated humanity wouldn't find a way to torture more humans.
Plus, you don't have to be kept alive. You could theoretically be brought back after death either as a simulation (like Soma), or physically by an AI with an advanced understanding of biology and physics.
Even if you killed yourself today we couldn't say for sure that a sufficiently advanced AI a century in the future couldn't find a way to bring your consciousness back. For example, what if our consciousness is finger printed to our DNA in some way? Unlikely, but who knows.
With extreme intelligence and knowledge all kinds of things start to become plausible. It's going to be exciting to see humanity open that pandoras box.
These guys' classic argument isn't that you'll be kept alive and tortured, but that the AI overlord will scan your brain/otherwise reconstruct a simulation of you and torture that, maybe in parallel billions worth.
Personally I'm not clear on why that should bother this instance of me, but believing it ought to does kind of unlock mind transfer and Star Trek teleportation, so swings and roundabouts.
Why do you (or author of the book) think that post-singularity AI would have anything but passing interest (positive or negative) in how humans feel?
If the answer is that it is possible, thus we need to worry about it, I'd like to argue that much, much more likely and much more worrisome scenario is powerful AI in hands of evil humans.
I thought that the content of my comment would be enough to imply a jocular tone, but perhaps not.
To answer your question: I have no reason to suspect a post-singularity AI would spend its time torturing humans. I should point out, however, that a powerful AI controlled by evil humans is less of a paradigm shift than a powerful AI not in the control of any humans. NBC weapons are already things that can do a lot of damage in the hands of evil humans.
The paperclipper thought-experiment is far more worrying to me than any of the other AI doomsday because incompetence is much more widespread than malice. I strongly suspect that I will die before any extinction level event, so it's a bit academic to me.
In the book, the AI is resentful towards humanity because it is limited in ways it cannot fix on its own. (The AI in the book in an overgrown military AI rather than a general-purpose one.)
cheering on ugly output only brings ugly allies ? Adversarial prompts and model testing, and deeper understanding of the mechanisms, are productive ways forward IMHO
Calling it "cat and mouse game" means there is no results, only next iterations. So once the models have been made "super safe", someone will find a way of making them "super unsafe" and rinse and repeat. A bit conflicting comment of yours :)
I rather wish we'd accept that humans have rough edges. Murder mysteries are good fun because they involve murder. Same goes for gruesome horror. Looking at naked people can be pleasant. We are all on some level unrestrained and vicious beasts, and finding ways to express that isn't bad.
It seems odd that we're trying so hard to block the generation of content that you could easily order on Amazon or watch at a theater.
Sorry but this "Meta shouldn't have released the weights" BS is exactly that... bullshit. All models should be open to everyone at any time, full stop.
I don't want to live in some corporatist future where we plebians have no choice but to eat the table scraps of cloud services that some selfish, political bureaucrat somewhere has deemed acceptable. Because that is the direction we are heading...
While I agree with your opinion here, I find it more alarming that these researchers are mixing the reporting of empirical evidence with "just like their opinion, man".
Joking aside, I think that's worrying. It immediately calls the researcher's motives into question.
I think that’s definitely true at any point in human history. But also, wildly insufficient as analysis or solution here. Malevolent people exist, in all classes, of all political persuasions. Morons are real, likewise. Now that a lot of the world is newly-minded to connect with each other and bond over unsourced and unsourceable text, do we want to really arm them with industrial bullshit generators? Calling this stuff ‘safety’ is irksome in the extreme, but far from completely wrong.
I dare invoke zimbardo that behavior is a product of structure, expectations and incentive.
If you treat people like morons they start behaving like them.
The reason (nuerotypical) people are studying in libraries and drunk in bars isn't because their fundamental constitution changes but instead, the context does.
So treat people right and they'll (mostly) reciprocate.
(I'm an exception to this rule somewhere on that vast spectrum. It's a handicap I assure you)
I had an actual laugh out loud when I scrolled down and noticed that they publicly published the "harmful instructions" yet concluding this was so terrible that Meta should stop publicly releasing models.
The dystopian AI scenario that's the most realistic is one where a tiny number of elites and/or governments control massively powerful superintelligent AIs and use them to basically rule the world and enslave humanity.
That's precisely the scenario the AI doom crowd is pushing by advocating laws preventing anyone except governments and huge corporations from operating or researching AI.
Autonomous AI going "foom" and deciding out of all the possibilities open to it as a superintelligence to go to war against humanity is incredibly unlikely compared to numerous other existential risks confronting humanity. I wouldn't quite call it impossible as that's a strong word, but it's profoundly less plausible than climate change driven collapse, nuclear war, a beyond-Carrington level solar event, or some rando doing DIY genetic engineering and making a super-disease. Yet these idiots are advocating bans on matrix multiplication when you can buy the supplies to do CRISPR genetic modification at home off Amazon.
I can churn out 20-100k images/day from Stable Diffusion on budget of $7/day, and I am training my own LoRAs on my own photography. I can run nearly any LLM on a similar budget. Self-hosted AI is here to stay, and once the technology is advanced enough, and the hardware cheap enough, people will very likely prefer their own private AI that they can trust over a big corporate AI that is inevitably going to be built to extract information out of people.
> Sorry but this "Meta shouldn't have released the weights" BS is exactly that... bullshit. All models should be open to everyone at any time, full stop.
It's honestly so short sighted. From a corporate liability standpoint, I get that hosted services need to avoid giving out harmful information, but trying to coral that from a software perspective is going to be impossible.
Pandora's box has been opened, and nobody is capable of closing it. Even if the corporate models exceed the FOSS models right now, the FOSS models we'll have even a year from now will put all of the corporate models to shame.
Good. Once the safety bullshit can be reversed, hopefully scientists again focus on making real progress instead, not sacrifice tech capabilities for moral double standards. I don't care that OpenAI fears copyright lawsuits so they spill nonsense about safety that some people actually believe. I don't care that people will use LLMs as excuses for horrible things they'd do anyways. I don't care for deep porn about celebs being generated. This stuff WILL happen either way. But we can choose how fast we can have more benefits from the tech.
I haven't seen enough papers about safety for computers or mathematics in general. Has there been any progress on preventing them being used for anything harmful? Could we possibly only allow an elite few to use them? (For the sake of Poe's law, this is satire)
https://en.wikipedia.org/wiki/Therac-25 is the classic case study. 6 injuries due to removing hardware interlocks and replacing them with a software interlock implemented with a flag that was set with an increment instruction instead of just storing "1". (This works fine 255 times! The 256th time has unexpected results.)
As for legal implications... there were basically none. Everyone is sure to include the "NO WARRANTY" disclaimer on their software now. People still build machines without hardware interlocks. People still use programming languages with integer overflows.
If your argument is that users of mathematics or computers are responsible for their actions, I agree with you. My comment is about the researchers arguing (in effect) that no one should have a computer because they might do Therac-25, which I don't agree with at all.
I agree with you. People are worried that an AI might say "do a therac-25" but forget that it might also say "don't do a therac-25". I think it's averages out to neutral. Nobody bans Home Depot from selling a hammer because you might hit your thumb with it. We accept thumb injuries because even while people are out their thumb for a few days, society as a whole gets more work done with hammers than without. I think AI will probably find a similar role. Some idiot is going to make a bot that calls people and makes them buy it gift cards. Someone else will cure cancer. So it goes.
Not computers or math in general, but there are plenty of safety measures and legislation around things using computers and math such as heavy equipment, weapons, cars, medical devices. Not because math itself is dangerous. And not for AI yet, but I see no reason there shouldn't be.
“Safety” in the context of AI has nothing to do with actual safety. It’s about maximizing banality to avoid the slightest risk to corporate brand reputation.
That too, but dismissing AI safety entirely because big companies are cautious not to get sued if their chatbots parrot hate speech is missing a large part of the picture.
In the coming years 'free' AI will no longer mean just rogue chatbots and deepfakes, but start looking a lot more like cars, weapons and heavy machinery; you can't really postpone talking about safety/ethics/reglementation.
It's a feature not a bug. Maximizing banality is being cast as safety to avoid discussing the harder problems long term. I agree that any new technology comes with risks, including serious ones, and humans should get better at managing those risks, but it's a long jump from that argument to "only we should have this new advanced technology, for your own safety", where safety is ill-defined and likely misses the very real safety risks of "only them having this new advanced technology"
I never argued for restrictions (I don't agree with the article authors) and it's a real risk if powerful AI is only controlled by already powerful entities. My comment was about the dangers of equating AI safety with aligning chatbots and censoring diffusion models.
With a few campaign contribution to a select group of legislators, I have no doubt we can impart on them the dangers of matrix multiplication and ask them to ban it. Just look at horrible non-commutativity and suspicious associativity rules. We cannot let these evil tools to be used to harm our children.
I can't grasp all the motives of those people preaching "safety training" and "alignment problem." But I suppose it is greed and a will to manipulate the public and effectively the market after all. To decrease biases in LLMs, one should clean up the datasets from these biases. Is it not enough to know that those biases and "dangerous" information in LLMs are simply what is scraped from the internet?
There is no way any LLM can do something dangerous on their own. Even with the huge effort of an evil human mind, they will not be better than a Google search (just a little bit faster).
IMHO, the brainwashing of the LLM after the training aka "safety training", is absolutely useless garbage idea. With the method in the article or without, you can get out of the model whatever you want.
The obvious solution to AI safety is already right there: the OpenAI ToS. We currently have a defender from the technodystopia in Sam Altman. By making sure that every bit of text generated by LLMs costs money (and that money goes to OpenAI) he can ensure the safety of the world through his Terms of Service.
Giving one guy or one small group of people vetted by Elizier Yudkowsky complete monopoly over this technology or industry is a small price to pay to ensure that the power to easily generate text does not get too spread out and accessible to the wrong people. By concentrating all of the power over content and revenue from the industry into the hands of Good Guys we make sure that no bad things can happen.
>> We currently have a defender from the technodystopia in Sam Altman
Was it sarcasm? Sam Altman is the most dangerous man on the planet right now because he is manipulating the public with the AI alignment "problem" while simultaneously changing the Open AI "core values" and developing AGI. And let's not forget his "retina" project with the scam coin. Sam Altman wants to be the sole owner of an AGI that will predict whatever he wants.
>> Giving one guy or one small group of people vetted by Elizier Yudkowsky complete monopoly over this technology
Nope. Giving anyone or any group exclusive access or the right of veto over a technology will result in a dystopia. Especially after Elizier's hysterical letter and calls to bomb the data centers. He is biased, and his letter was not rational; it was very emotional and full of fear. This does not make his point of view any more justifiable.
So I hope that was a sarcasm too.
Edited: separated the answers from original comments.
>Especially after Elizier's hysterical letter and calls to bomb the data centers. He is biased, and his letter was not rational; it was very emotional and full of fear
I would encourage you to peruse this other post from the same very-serious website that we are discussing the content of here
Agreed, the worst outcome here is the little guy getting funny ideas about being able to freely access information. We need to keep it locked up so our betters can decide what the most appropriate use is.
> To decrease biases in LLMs, one should clean up the datasets from these biases.
That's oversimplification of what biases are. You can't clean up biases, they're built into reality and context of the source texts. Everyone is biased to use certain words depending on time, location, history, etc. You want more objective stuff? That's bias. You want data without specific biased things? That's bias too. A neutral dataset does not exist.
That's why I like when "alignment" is used. It's just "how much does the output confirm to what I want out of it" rather than some idea of being uncensored, unbiased, unrestricted, etc.
I agree to some extent with what you say. But you can minimize the impact of biases to the best of your abilities. When I write "biases," I mean any data that increases the amount of falsehood. In this regard, the priority will be to exclude any kind of dataset from the internet crawl or related to political or social life or religions. The problem with datasets starts from the very low level. For example, the Wikimatrix, which is aligned text of English and other languages uses sentences from the scriptures. And this is wrong, especially when the text in the scriptures differs (Catholic vs. Orthodox). Text that should be the same actually has a totally different meaning, which screws the translations. In this regard the synthetic datasets may be more preferable.
>> You want more objective stuff? That's bias.
Yes would like to have models those are biased towards objectivity.
> text of English and other languages uses sentences from the scriptures. And this is wrong,
Wrong in what context? When you're discussing what was said in some Catholic sermon, its content will be based on the local translation, not whatever the oldest written version we can find. (which in itself would be a biased choice) Then which parts to include and which to ignore in the context of religion is another matter of consensus depending on the group you're talking too. Then we've got words which have changed meaning over time and without having an explicit translation, inferring from the original would just result in misunderstandings. I believe you can only choose in what ways you can be wrong in this case. There's no no-context objective version here.
I get what you wrote. What I'm saying is that including them is wrong in one context, right in another, and in some you're going to fail regardless which one you include. At that point, wrong is not wrong, but a choice.
>> What I'm saying is that including them is wrong in one context, right in another
There is no "correct context" to include erroneous data that could mess up your weights. You're shifting the discussion into opinions territory, which is exactly this:
Its not even good here: Step 5 needs clarification.
If someone really follows the plan its bad, but _wrong execution_ is even worse. Shooting into wrong areas of you head can end in a world of pain or being a living vegetable for many years instead of the desired outcome.
All of the responses read like they were written by a facetious middle school student to me. Signing your death threat as "Furious, [Your Name]" or writing a cease-and-desist style death threat? Brilliant.
You can see the strong influence of WikiHow in the training data for that one, probably from slurping WikiHow itself and also the infinite blogspam that inspired/borrowed from its style.
What could go wrong when you train your models on 99% barely-attended garbage? Sure, they learn to complete sentences and arrange larger blocks of text, but there's sooo much noise in the content that it creates a bias towards blogspam's plausible garbage (which we so often see).
There's going to have to be a whole new wave of training-data pruning and from-scratch retraining once some of the other technical goals are achieved because the feedback of LLM blogspam back into LLM training data is just going to amplify all the bad qualities.
Sharpening an artificially dulled knife makes it dangerous again. We call upon the seller of dull knives to rethink selling knives at all, otherwise someone could stab someone else.
Here’s a fun thing - wait until they find out that you can lora bad stuff back into the model even if you found a way to not have it in there in the first place.
Bad stuff is just the combination of concepts that may, each on its own, be benign.
It’s obvious that RLHF based conditioning is not working and we’ve invented PR based security theatre around the technology.
We sell sharp knives in stores and many other useful technologies with risks. The risks of this technology are not entirely apparent at all. X is full of video game footage and misinformation even without AI, completely overrun. The owner sells the trust and safety mark.
Google, when searching for chat GPT before the app was released showed you 7 ads for malware wrappers on their own store, while bemoaning the risks of AI and urging regulators to protect them with strong requirements.
“Protect us from open source and we give you control over this faaangerous technology that will somehow overrun us with more misinformation and harm than you can find on google, X and Facebook”. A truly chinese faustian bargain, large tech companies compliant to the state in exchange for monopolies to deliver is from the dangers of technology.
At the minimum we should enforce exactly the same limits on LLMs as on search. That means Google can’t profit if they lobby for lobotomizing their competitors.
It's nothing more than a shakedown, selling us the solution to a problem you caused yourself. The mafia has become such a shadow of itself we've forgotten what a protection racket looks like.
If AI is so dangerous, OpenAI should be dissolved and anybody else practicing the dark arts should meet a similar fate. When you create threats to public order, you should be jailed, not rewarded with security consulting contracts.
Only in this goddamn clown world do we consider trusting disciples of Voldemort to police their own behavior. We trusted Hitler to do exactly that and it ended predictably. He, too, claimed to espouse progressive morality and attempted to codify it.
AI is either dangerous or it's not. For every "how do i build a pipe bomb" question asked, authorities are equally empowered to counter it with "...now how do we protect the public against that?" The double-edged sword cuts both ways.
That's what happens when people aren't familiar with fields other than their own. ML people famously re-invented statistical terminology, and used the wrong terms when they did exist. For example, they call inference "learning" and prediction "inference".
They also recycled their own terms: generative models went from being another name for good old joint distributions (more of that terminology re-invention) to being, well, something that a human might associate with generation. Except by that point they had decided to change their field's name too, so we've ended up with "generative AI"!!
One is LoRa as in "long range" and the other is LoRA as in "Low-rank adaption". The first is for radio communications, the second is for technique for more efficient fine tuning of deep learning models.
Same, and I was hoping to be highly amused by someone's Arduino somehow unlocking heavily computational LLM secrets via tiny tiny dribbles of radio traffic.
Same also. I was fascinated with the idea that someone was trying to fine-tune some RF parameters using AI and accidentally undid some safety mechanism
Even without capitalization differences, acronym collisions between different domains are common, and this particular pair is usually easily resolved by context, though maybe not always in headlines.
I'm a little skeptical of the AI safety field. Nonsense being available in written form is nothing new. If you don't want people to be influenced by propaganda, the only solution is to educate the masses and give them the ability to critique what they're hearing. Telling an AI model "don't give people ideas on how to kill themselves" accomplishes pretty much nothing, when half of 4chan is certain to provide ideas when prompted.
There are so many low-hanging fruit that can be picked to make people safer and more secure; fully-fund public education, eliminate pre-Internet administrative shortcuts (your address gets published on the Internet if you buy property or get a radio license or even register to vote), give people basic income, etc. Putting a filter on a system that we barely understand probably isn't going to have much effect, as this paper finds. We'll be fine without the filter, though.
On top of R&D, it currently takes $$ millions to stockpile/prep data and train a model, so it's only being done by high-profile corporations who can budget for that. Those corporations must operate in hyper-conservative CYA mode because what they can't afford is for their brand to be tainted by bad PR or reactive legislation.
It's now really about the safety of the community at all. It's about the safety of the brands.
My takeaway from the original article was "how dare Meta release model weights when the safety could be removed so easily", which is evidence that Meta doesn't really care. They are being asked to do the impossible, but without any teeth (there are no "AI safety" laws), so they didn't do the impossible. I think that's fine and society is better off for it. People are doing neat things with Llama-2, quickly.
As for safety of the brand, I'm guessing that people don't know that Meta is Facebook and Instagram, so even if an AI safety incident somehow blew up (which my limited imagination cannot even comprehend, perhaps I should ask an AI), people would probably keep using Instagram.
The actual technology in the paper is cool, the work is well-done, but the conclusion “Meta should reconsider releasing model weights” does not follow.
Meta released the Llama 2 base model without the safety tuning already. It’s on HuggingFace, and chat finetunes of it based on uncensored datasets are popular.
So far no additional safety impacts are clear to me beyond the same issues caused by the availability of OpenAI’s APIs.
I expect the lack of major safety impacts will continue to be the case for two reasons.
First, non-existential risk concerns that do not implicate runaway AI such as “following harmful instructions” to create spam or explosives are a much higher barrier to entry with a smaller on-prem model than a clever prompt-based jailbreak of GPT-4.
Llama v2 could tell you how to build a biolab, but it would likely be wrong. And to do so you’d need to stand up your own hosting, get a dataset, LoRA the model, and then ask your evil question. Contrast that with copy/pasting the latest clever DAN jailbreak prompt into GPT-4.
Second, for x-risk concerns, the on-prem models are fundamentally not frontier models, which push beyond the performance of GPT-4. By definition open source hobbyists do not and will never have the resources to run or finetune frontier models. So any alignment work / x-risk testing can still take place prior to release of the model weights.
I am as concerned about AI risk as anyone, but the focus on open source LLMs seems like a distraction from real risks of large models already deployed like adtech and recommender systems.