Remember when the world freaked out over encryption, thinking every coded message was a digital skeleton key to anarchy? Yeah, the 90s were wild with the whole PGP (Pretty Good Privacy) encryption fight. The government basically treated encryption like it was some kind of wizardry that only "good guys" should have. Fast forward to today, and it's like we're stuck on repeat with open model weights.
Just like code was the battleground back then, open model weights are the new frontier. Think about it—code is just a bunch of instructions, right? Well, model weights are pretty much the same; they're the brains behind AI, telling it how to think and learn. Saying "nah, you can't share those" is like trying to put a genie back in its bottle after it's shown you it can grant wishes.
The whole deal with PGP was about privacy, sending messages without worrying about prying eyes. Fast forward, and model weights are about sharing knowledge, making AI smarter and more accessible. Blocking that flow of information? It's like telling scientists they can't share their research because someone, somewhere, might do something bad with it.
Code lets us communicate with machines, model weights let machines learn from us. Both are about building and sharing knowledge. When the government tried to control encryption, it wasn't just about keeping secrets; it was about who gets to have a voice and who gets to listen. With open model weights, we're talking about who gets to learn and who gets to teach.
Banning or restricting access to model weights feels eerily similar to those encryption wars. It's a move that says, "We're not sure we trust you with this power." But just like with code, the answer isn't locking it away. It's about education, responsible use, and embracing the potential for good.
Innovation thrives on openness. Whether it's the lines of code that secure our digital lives or the model weights that could revolutionize AI, putting up walls only slows us down. We've been down this road before. Let's not make the same mistake of thinking we can control innovation by restricting access.
The fight against encryption continue to this day and while https is now ubiquitous, large-scale cdns makes it somewhat a moot point and emails are still largely plaintext.
But people's private digital communications have largely moved to platforms like WhatsApp and Messenger which enjoy end-to-end encryption. Email, at least between major providers, today enjoys TLS over the wire while being sent.
I'm sure there are various flaws and weaknesses and maybe even backdoors, but trying to make it sound like we lost the fight for encryption because emails are in plaintext is rather disingenuous.
I'm not sure Facebook and a now Facebook owned platform are good examples for private communications. There was an article posted here a week or two ago detailing how Facebook sold access to the contents of users private messages to advertisers.
It represents a step forward from the 90s for the vast majority of people. E2E in messenger and WhatsApp is still painful for LEO.
The article last week (assuming you're referring to this [1]) involved users consenting for Netflix to see their messages. A user from the 90s could have made the same mistake sharing plaintext emails.
I share your concerns and think you're broadly correct. I think it's worth adding some nuance though.
When you drill into specifics there are almost always exceptions. For instance, in your example about sharing research, there are certainly some types of research we shouldn't automatically and immediately make publicly available like biological superviruses or software vulnerabilities.
I think the same can be said about AI. We should aim to be as open as we can but I'd be hesitant about being an open source absolutist.
I agree we want to democratize AI and we should be very, very weary of powerful people trying to get a monopoly on AI.
But I'm not ready to say absolutely everything should immediately be shared with everyone. At least not until it's clear we know what we're dealing with.
I know it can be hard to trust people but the fact is we have to. Even today, there are many people with the power to end the world (nuclear weapons, viruses etc) but we trust the people who have these capabilities not to abuse their power. We do this because we don't want anyone to have the right to launch nuclear weapons. And I think that's wise.
I definitely don't know for sure but AI may be another one of these technologies.
Either way, you can fight against regulatory capture, the downsides of capitalism etc without being an open source absolutist.
Weights is derivative work, and as such should follow the licensing of the original works. If those works are public domain or were appropriately licensed, distributing weights openly should be protected as free speech.
And let’s not anthropomorphize of ML. Models don’t “think” or “learn”. The only party with free will and agency is whoever makes or operates them; trying to paint an unthinking tool as a human is just a means of waiving responsibility for them.
The concept that you call “dodgy” is actually fundamental to copyright. If you remove the idea of derivative works, the entire system that encourages innovation by protecting intellectual property (yes, the same system that gave us computing and ML) falls apart.
The one that encourages innovation by allowing copyrighted works to stay copyrighted for 100+ years? That's a long time to wait for something to build on something!
Unfortunately nobody told Marvel, Disney, and the Tolkien estate about it, so we'll have to wait until we're all dead before being able to build ontop of the cultural works that defined our childhoods.
"Something" sure. But I can't make something set in Middle Earth (I can't even say the word hobbit, ask D&D how that went), I can't write and sell my own Spiderman comics, I'd have a hard time writing my own Cinderella story without Disney breathing down my neck.
This is a very different reality from the "We’ve been building on top of each other’s work in the open-source software ecosystem and in science this entire time".
You are trying to pass off some arbitrary thing for “building on top of”.
I can use all the benefits of Linux to deliver a SaaS (like 99% of us do), I just can’t call my SaaS “Linux Something”.
You can use all the ideas Tolkien used in your own creative work. You just can’t call them things he called them for now.
> I can't make something set in Middle Earth (I can't even say the word hobbit, ask D&D how that went), I can't write and sell my own Spiderman comics, I'd have a hard time writing my own Cinderella story without Disney breathing down my neck
There is no creative reason to do those things. The only reason is to commercially profit by piggying back off the big names. Copyright works as intended.
for now, the last 80 years, and until easily 2040.
> I can use all the benefits of Linux to deliver a SaaS
> You can use all the ideas Tolkien used in your own creative work. You just can’t call them things he called
That's quite a bait and switch. You don't have to rewrite a linux kernel, or gcc, or whatever language you wish to use in order to make your SaaS. You aren't limited to using the "idea" of a linked-list, a hashmap, or HTTP, and have to reimplement it yourself from scratch. But that's exactly what you're proposing for literature.
I can't build on the idea of Spiderman by making ArachnidMan, a crimefighting superhero who got pinched by a electromagnectic spider without wondering "Is today the day Marvel sues me into the ground". And I absolutely cannot write my own Spiderman comics.
> There is no creative reason to do those things. The only reason is to commercially profit by piggying back off the big names. Copyright works as intended.
In the US at least, copyright comes from a clause which states that laws can be passed "to promote the Progress of Science and useful Arts". The only thing the current system is promoting is the concentration of copyright into large corporations (see music labels) and locking people out of the cultural artifacts that define their life (Disney gets to take public domain stories, but a slight twist, and copyright them for 100+ years).
What there is no creative reason is the way that the system is currently set up. All that exist as blatant money and power-grab reasons.
> for now, the last 80 years, and until easily 2040.
Let it be until the end of the universe, please. What kind of creative are you anyway if you don’t want to do your own world-building?
> You don't have to rewrite a linux kernel, or gcc, or whatever language you wish to use in order to make your SaaS. You aren't limited to using the "idea" of a linked-list, a hashmap, or HTTP, and have to reimplement it yourself from scratch.
That was exactly to show you how copyright doesn’t prevent us from building on top of things, but instead fuels innovation. IP protections is how one guy can say “you must always provide your source if you use my work” and have it quickly grow into a massive ecosystem on which most of today’s Internet runs to this day.
Open-source licensing can only exist thanks to copyright and the idea of derivative works in particular.
1. In order to license something, you have to be able to enforce it legally, and for that you have to be the author, which is what copyright means.
2. In order to encourage others to use your library but also ensure they contribute back if they make changes, you need to talk about the concept of a work that is based on your work, and that is—drumroll—a derivative work.
> "to promote the Progress of Science and useful Arts"
If you are saying that allowing any writer to be able to take the world someone else built and profit off its fame with minimal modifications is somehow promoting progress of the arts, then I don’t know what to say to you.
> What kind of creative are you anyway if you don’t want to do your own world-building?
A no true creative.
What kind of creative needs lifetime + 70 years of monopoly on an idea?
> If you are saying that allowing any writer to be able to take the world someone else built and profit off its fame with minimal modifications is somehow promoting progress of the arts, then I don’t know what to say to you.
I am not. I am saying that the current way it's implemented is broken. Tolkein has earned more than enough money off of LOTR, it's far past the time for it to enter the public domain and join the stories that he build his work on.
Many of Disney's earlier movies are retelling of folktales. Tolkein in addition to creating parts of his world build on top of existing folktales. Neither would be where they are now if someone was able to impose the kind of restrictions on them that they themselves now impose on us.
I don’t know whether Tolkien’s estate should or should not keep profiting from his work, but I don’t see why this should be forbidden. If anything, we probably have so much great work coming out because people want to repeat the success of great masters and achieve fame and wealth that are possible thanks to IP protections.
Make a cool world and license its use semi-openly, be the change you want to see? Just keep in mind that if your world is openly licensed, it will quickly go out of your control and likely in a direction you (or your family, when you’re dead) may find repulsive. Also, I think you would have to be comfortable that if the next guy writes something based on your hard work and gets a Netflix deal tomorrow you may find it difficult to do something about it.
> Many of Disney's earlier movies are retelling of folktales. Tolkein in addition to creating parts of his world build on top of existing folktales. Neither would be where they are now if someone was able to impose the kind of restrictions on them that they themselves now impose on us.
Anything you say, write, do, etc., is technically based on everything done before you. So now we should eschew the idea of intellectual property and the progress it brought. Got it!
> I don’t know whether Tolkien’s estate should or should not keep profiting from his work, but I don’t see why this should be forbidden.
Because you and I are arguing, or at least I am arguing, about the duration over which they should keep profiting off of it. I claim that it's self evidence that a copyright that never expires and is transferable is self evidently bad (companies will end up having monopoly rights to every idea, see their concentration of money).
And I further argue that the current length for copyright also problematic. the current life of author + 70 years is ridiculously long time for a monopoly. It's gotten to that length not because starving artists need it, but precisely because unimaginative large corporations found it easier to extend the copyright on their existing IP than to image something news.
> Make a cool world and license its use semi-openly, be the change you want to see?
But even if you choose not to because you wish to monetize your work, that's fine by me. But lifetime + 70 years is too long of a time.
My thought is that we should have it be non-transferable, no more having authors and musicians sign their copyright away for a pittance. At it should be capped at something like 30 years or 20 million dollars, whichever comes first.
Gives people time to monetize their idea, and if 30 years wasn't enough time, no amount of time will be enough because nobody's looking at your work. And the amount of works that earn over 20 million dollars is a rounding error; you get to be rich, and then the rest of us get to remix your work.
> Just keep in mind that if your world is openly licensed, it will quickly go out of your control and likely in a direction you (or your family, when you’re dead) may find repulsive. Also, I think you would have to be comfortable that if the next guy writes something based on your hard work and gets a Netflix deal tomorrow you may find it difficult to do something about it.
Both of these I find acceptable. Firstly it's my work only in the sense that I was the one who made it, not it the sense that it belongs to me. Copyright doesn't mean that I own the idea, it only gives me a monopoly on monetizing and distributing the idea.
Secondly, this is exactly how open source projects work. The parallel you drew earlier to open source work comes back here. You have no control over what people make or do with your open source licensed project, all you enforce is what requirements they have to share the code (this quickly becomes a semantic discussion).
The GPLv3 and APGLv3 licenses, which people derogatorily label as "viral", does not even attempt to limit you from making money off the code, all it does is require you to allow everyone else to access the changes you make to it and code you built off it.
The profit cap is an interesting idea in theory, maybe it would discourage some people who go into it hoping to make infinite money but maybe that is for the better.
Regarding software, true, we can’t limit what people build with it if you open-source it, we sort of do not think about it (I wonder for how many people it is an issue secretly).
I think it is similar in art, however. Fanfics are generally tolerated, unless you do something the author specifically dislikes or try to compete with the original. In Japan I think it is also legal to sell them (see dōjinshi) unless the copyright holder specifically complains (which they do not tend to). But using Middle Earth in a commercial ad for a car, for example, is insta-lawsuit.
> maybe it would discourage some people who go into it hoping to make infinite money
That's totally fine by me, the point of copyright should be to advances the arts and science, not to help concentrate wealth.
> I wonder for how many people it is an issue secretly
I think quiet a few. See for example ElasticSearch and Amazon. If you want a truly open-source license then you have to accept that. Otherwise there's a whole spectrum of "source available" licenses, or something like CC BY-NC-SA.
Though that's another difference between works of art and source code. Releasing my book doesn't produce a maintenance burned on me, an open source project usually does.
> using Middle Earth in a commercial ad for a car
You don't even have to go as far as a car, see [0]. They didn't get sued, but it's safe to say that they would the moment they try selling it.
As for the other aspect of fanfic, I think the copyright's holders desires should be weighed less. I personally disliked many of the star wars movies, particularly the latest trilogy, and I'm not alone in this.
A) I know for a fact that better star wars movies would be possible if Disney did not hold a death grip around the IP by the simple fact that other good movies can be made, there's nothing intrinsic to star wars.
And B) just as copyright holders won't stand by if you try to do a fanfic that they don't like, I shouldn't have to be forced to stand by and watch them butcher (yet again) the stories that I grew up on AND be told that that's the only version available.
Painting something you dislike as “against nature” is the lazy way out and not a real argument.
Humans are part of nature. Property rights are necessarily part of nature too, by extension. It can’t go “against” the flow of nature when it is the flow of nature.
Derivative works are only concerned with the licensing of the original work if they don't fall under fair use. It's really, really hard to argue that an open weight AI doesn't fall under fair use.
1. "the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes;" -> an openly distributed model is not of commercial nature. Check one for fair use.
2. "the nature of the copyrighted work;" -> Harry Potter and The Sorcerer's Stone has very little in common nature with an artificial intelligence that produces arbitrary text based on a prompt. Check two for fair use.
3. "the amount and substantiality of the portion used in relation to the copyrighted work as a whole;" -> The amount (all of it) goes in favor of the original work, but the substatiality of it is definitely more ambiguous, as while if they had used none of the copyrighted works, the AI couldn't exist, any single original copyright holder's work contributed very little to the end result.
4. "the effect of the use upon the potential market for or value of the copyrighted work." -> No one's using AI to avoid having to buy a copy of Harry Potter. An AI and a book are not currently considered substitutable goods.
Basically, even if a model's weights constitute a derivative work of the training material's copywritten material (that's another can of worms, there's a finite possibility copyright doesn't protect/apply to weights at all), there's a very good argument to be made for fair use.
1. First, there are commercial models. But also, we could consider that if something specifically enables unfair use then that may deserve a look.
2. Weights are a derivative of another nature, but one that specifically enables mass production of derivatives of the same nature. Could be an important concept that simply didn’t exist until now.
3. That’s roughly my understanding, too.
4. Valid point on one hand, but consider that using LLMs is exactly substitutable for any non-fiction book or resource. If no one can sell a book, then who would write them?
Not really - "arguing over where to draw the line" would be a discussion about how a hypothetical new law should treat model weights; however, assertions about whether they are a derivative work (or even "work" as such, in the sense of copyright law - noncreative transformations aren't copyrightable works, no matter how much "sweat of brow" or money they took) is a discussion about where the old copyright laws have drawn the line, and the "should's" don't matter there, it's about how the courts will interpret the precedent, and at the moment they haven't said that they are derived works, and while they could declare that, there are all kinds of arguments why they also might declare the opposite.
Off-topic but this user seems to be using ChatGPT or something similar for almost every single comment. Does Hacker News have a stance on this or is the thinking that it is allowed as long as the content is good?
If posts with multiple paragraphs encircling a topic seems suspect, then we're all guilty. Their points are cogent. If they've dressed the arguments in robes of "purple", so be it.
Maybe we need to up our game then. That could very well be mergekit and clever prompting. Their comment history bends in the direction of content engineering and will get less discernible the more accounts are doing it.
Please, these kinds of purple-prosed thesaurus-laden ruminations smack of LLM augmentation - two in the same thread, and neither contributing anything much more than a long winded flowery restatement of the existing state of the discussion:
It's important not to jump to conclusions about how others are participating in discussions, especially based on the content or style of their comments alone. Many users might have a consistent way of expressing themselves, or they may be leveraging various tools and resources to enhance their contributions.
The key focus should always be on the quality and relevance of the content shared.
This comment reads like a GPT-generated comment! When I look at your history, your other comments seem genuine. I’m probably totally wrong, but I do like the idea of using GPT to counter anti-GPT sentiment. :)
That doesn't only feel odd, it also is empty prose based on a far-fetched analogy. Apologies to the root commenter if they wrote it themselves, but quality and relevance is lacking here. 99% sure some of their posts are at least heavily augmented.
I don't think this analogy is wholly applicable, simply given the scale and potential blast radius of certain classes of models. A more apt analogy would be nuclear technology. There are the Atomic Gardeners on the one hand, who believe in only the good and see the promise of the technology and peoples intent, and then there is the bitter struggle for power and threat which hinges on it.
In most cases, ML models have the capability to revolutionise or at least augment/optimise problems in predictable ways. In extreme cases, deepfake technology and the like can erode at the tenuous levels of trust which hold societies together. We have seen what happens when disinformation, mistrust, and even basic levels of technology meet: look at students being lynched in India, Pakistan, and elsewhere due to WhatsApp group messages claiming blasphemy; or the practice of SWOTing; or the insinuation of CSE gangs leading to a pizza parlour in the US being stormed by someone with a machine gun.
The stakes here are a lot larger than pure technologists, todays Atomic Gardeners, may perceive.
In that context, legislation is trying to provide a counterweight to the pace of change to allow for some – any – breathing room, and particularly to prevent an increasingly hostile cast of nation states from weaponising that technology, even in small ways. For example, OSINT of North Koreas operations show that the basic parts needed for making baby powder are also capable of being used for weapons development.
I think all the issues that you've listed are serious. They're the things that keep me up at night. But I don't think the content is really problematic (except maybe deep fakes), it's the untraceable nature of their sources. To me, it's anonymity that gives them their power.
If we knew that the content was posted by a specific REAL person/company/government, we could judge it's authenticity and hold them liable if it's criminal in some way.
Not saying there isn't a place for anonymity on the internet but to me, that's the problem that needs to be solved more so than AI.
imho, and you can call me an atomic gardener if you wish, nuclear tech is a bad example. If we had actually gone "all in" on nuclear to produce electricity, our and the world's clean energy footprint might result in warming of entire degC's less than it's currently projected to be. Climate change is a global problem -- what stakes could possibly be higher than ecological collapse?
I actually cannot think of any example where successfully withholding technology has helped people thrive and prosper. People using this tech to sow distrust is outweighed, so obviously and so massively, by the benefit of having an intelligent machine companion under their complete control.
Every "weapons technology" in human history has been used for good. Every single one.
Give every script kiddie, bored teenager, Mexican cartel and scammer an AI that can mimic anyone's voice and likeness, and the world will get a lot messier.
I don't think they're the same. I wish we could put the genie back in the bottle. I think AI will make humanity less special.
I'm not convinced society will be better with AI. The benefits must push down cost of living for the masses, improve quality of life for the masses, all without destroying society with disinformation and shattering job loss.
I think LLMs have considerably more probability to make the world worse overall than cryptography does (the sheer level of information bullshit they can develop for pennies is going to transform our society and I doubt it is going to be for the better). Still I don't see the point of banning open weight models and LLMs that don't have guardrails. And I'm not sure you can realistically construct laws that would do it accurately. The genie is out of the bottle, pandora's box is opened, etc, etc. And locking down models with guardrails is only something that corporations have to do in order to avoid having a public racist chatbot problem and the associated headlines.
Life, all biological life with us as a kind of pinnacle, is about to go through radical change.
There is no risk free path. It isn’t guaranteed that a single human will be alive in 100 years - because we failed, or even because technologically we succeeded
But a degree of openness is necessary for our best ideas, our most good faith collaborations, to have a chance
It is more chaotic to trust each other, en masse. But I also think it is our best bet
> Just like code was the battleground back then, open model weights are the new frontier. Think about it—code is just a bunch of instructions, right? Well, model weights are pretty much the same; they're the brains behind AI, telling it how to think and learn. Saying "nah, you can't share those" is like trying to put a genie back in its bottle after it's shown you it can grant wishes.
I think telling a genie "I wish for no more wishes" is a common enough trope.
I'd agree that making weights available is basically irreversible; however making it illegal to make new sets of weights available is probably fairly achievable… at present.
-
Some of the issues with "winning" the battle for encryption include:
1) We also need it to defend normal people from attackers
2) It's simple enough to print onto a T-shirt
3) The developers recognised the value and wanted to share this
The differences with AI include:
1) The most capable models don't fit on most personal devices at present, let alone T-shirts
2) 95% of the advantages can be had from centralised systems without needing to distribute the models directly to everyone
3) A huge number of developers have signed an open letter which is basically screaming "please regulate us! We don't want to be in an arms race with each other to make this more capable! We don't know what we're doing or what risks this has!"
As long as everyone admits that number 3 “please regulate us” is really just asking for regulatory capture and is in no way a good faith move, but rather protectionism. Then I’m good to proceed with these conversations.
These people have not just suddenly developed consciouses. This is a game move.
1) You can run a very large model on existing cheap hardware, with poor performance, but it will run. Also, "most personal devices at present" is doing all the work here. Obviously if there is now demand for devices with hundreds of GB of VRAM then sellers will soon make them and buyers will buy them. That amount of memory is not actually that expensive. And it's presenting a false dichotomy where the only alternatives are companies the size of Microsoft and the ability to run GPT-4 locally on your existing iPhone. There are thousands of medium-sized companies that can afford five to six figures in hardware, and any individual could rent big hardware on any cloud provider -- requiring them to compete with each other -- instead of locking each model behind a single monopoly service.
2) Claiming that the benefit of people being able to run the models locally is only 5% is absurd. The privacy benefit of not sending your sensitive data to a third party is worth more than that by itself, much less setting up some huge institutions to have an oligopoly over the technology and subject everyone to all the abuses inherent in uncompetitive markets. But from the perspective of those institutions they want to classify those benefits as costs, so they try to come up with some malarkey about how they need to gatekeep for "safety" because so they can control you is unsympathetic.
Ah, now I see my mistake; I gave a list of what I see as things of political importance without making it clear this is what I was doing.
That the source code for unbreakable encryption fits onto a T-shirt made it clear to the US government that it was a "speech" issue, and also made it clear that it was in any practical sense unstoppable.
That big models can't fit onto most person devices at present is indeed likely to be a temporary state of affairs, however it does also mean that the political aspect is very different — the senators and members of parliament can't look at the thing and notice that, even if the symbols are arcane and mysterious beyond their education, it's fundamentally quite short and simple.
And, of course, if they're planning legislation, they can easily say "no phones with ${feature} > ${threshold}", which they've already done on various compute hardware over the years, even if the thresholds seem quaint today: https://www.youtube.com/watch?v=l2ThMmgQdpE
Politicians don't care if all of us here think they look silly.
> 2) Claiming that the benefit of people being able to run the models locally is only 5% is absurd. The privacy benefit of not sending your sensitive data to a third party is worth more than that by itself, much less setting up some huge institutions to have an oligopoly over the technology and subject everyone to all the abuses inherent in uncompetitive markets.
This seems a surprising claim, given the popularity of cloud compute, Cloudfare, third-party tracking cookies, DropBox, Slack/MS Teams, Gmail, and web apps run by third parties such as Google Docs, JIRA, Miro, etc.
There are certainly cases where people benefit from local models. The observation of all these non-local examples is part of why I think that benefit is about 5% — though perhaps this is merely sample bias on my part?
> But from the perspective of those institutions they want to classify those benefits as costs, so they try to come up with some malarkey about how they need to gatekeep for "safety" because so they can control you is unsympathetic.
They've been saying this "malarkey" since before they had products to be marketed.
> That big models can't fit onto most person devices at present is indeed likely to be a temporary state of affairs, however it does also mean that the political aspect is very different — the senators and members of parliament can't look at the thing and notice that, even if the symbols are arcane and mysterious beyond their education, it's fundamentally quite short and simple.
You can write the RSA algorithm on a t-shirt but in practice you need a computer to run it.
Likewise, you can fit llama or grok on a thumb drive and carry it around in your pocket, but in practice you need a computer to run it.
> And, of course, if they're planning legislation, they can easily say "no phones with ${feature} > ${threshold}"
But what good is that?
If they're actually trying to accomplish an outcome, those kinds of rules are completely useless. If all they're trying to do is check the "we did something about it" box, there are a hundred other useless things they could do that would be equally ineffective but still check the box and cause less collateral damage.
> This seems a surprising claim, given the popularity of cloud compute, Cloudfare, third-party tracking cookies, DropBox, Slack/MS Teams, Gmail, and web apps run by third parties such as Google Docs, JIRA, Miro, etc.
Notice how these are the kinds of things that major companies with trade secrets to protect have explicitly banned, and individual consumers with no bargaining power suffer while complaining about it?
> They've been saying this "malarkey" since before they had products to be marketed.
Eccentrics have been saying this "malarkey" for a long time. It becomes the official position of the market leader when it is convenient.
> You can write the RSA algorithm on a t-shirt but in practice you need a computer to run it.
Pretend you're a politician: the thing you've been told is a state secret is written onto a T-shirt by a protestor telling you it isn't secret, everyone already knows what it is, you're only holding back the value this can unlock. You, the politician, might feel a bit silly insisting this needs to remain secret even if you think all the 'potential value' talk is sci-fi because you can't imagine someone doing their banking on the computer when there's perfectly friendly teller in the bank's local office.
If the same discussion happens with a magic blob on the magic glass hand rectangle which connects you to the world's information and which is still called a "telephone", you might well incorrectly characterise the file that's actually on the device of the protestor talking to you as "somewhere else" and "we need to stop those naughty people providing you access to this" and never feel silly about your mistake.
> But what good is that?
"Then the people can't run the 'dangerous' models on their phones. Job done, let's get crumpets and tea!" — or insert non-UK metaphor of your choice here; again, I'm inviting you to role-play as a politician rather than to simulate the entire game-theoretic space of the whole planet.
> cause less collateral damage
I made that argument directly to my local MP about the Investigatory Powers Act 2016 when it was still being debated; my argument fell on deaf ears even with actual crypto, it's definitely going to fall on deaf ears when it's potential collateral damage for a tech that's not yet even widely distributed (just widely available).
> Notice how these are the kinds of things that major companies with trade secrets to protect have explicitly banned, and individual consumers with no bargaining power suffer while complaining about it?
No.
Rather the opposite, in fact: each is used by major companies.
> Eccentrics have been saying this "malarkey" for a long time. It becomes the official position of the market leader when it is convenient.
It was the position of OpenAI with GPT-2, which predates their public API by 16 months, and them being a "market leader" by 3 years 9 months:
February 14, 2019: "Due to our concerns about malicious applications of the technology, we are not releasing the trained model. As an experiment in responsible disclosure, we are instead releasing a much smaller model for researchers to experiment with, as well as a technical paper." - https://openai.com/research/better-language-models
June 11, 2020:
"""What specifically will OpenAI do about misuse of the API, given what you’ve previously said about GPT-2?
With GPT-2, one of our key concerns was malicious use of the model (e.g., for disinformation), which is difficult to prevent once a model is open sourced. […]
We terminate API access for use cases that are found to cause (or are intended to cause) physical, emotional, or psychological harm to people, including but not limited to harassment, intentional deception, radicalization, astroturfing, or spam, as well as applications that have insufficient guardrails to limit misuse by end users. As we gain more experience operating the API in practice, we will continually refine the categories of use we are able to support, both to broaden the range of applications we can support, and to create finer-grained categories for those we have misuse concerns about.""" - https://openai.com/blog/openai-api
People called them names for this at the time, too, calling their fears ridiculous and unfounded. But then their eccentricities turned out to lead to successfully tripping over a money printer and now loads of people interpret everything they do in the worst, the most conspiratorial, way possible.
I'm reminded of the last Narnia book, specifically the Dwarfs who could not see their surrounds.
"""They have chosen cunning instead of belief. Their prison is only in their minds, yet they are in that prison; and so afraid of being taken in that they cannot be taken out."""
Is it really so hard to believe they might be sincere? We've had tales of scientists — and before them alchemists and wizards — undone by their creations for a very long time now. It's a very convenient meme to latch on to; surely even if you do not share this caution, or find it silly for whatever reason, you can at least be aware of it existing in others?
The term "conservative" may be now affiliated with a specific political team more than the concept of being risk-adverse, but the idea is found even in the history of the ancient Romans and the tales from ancient Greece.
And I say this as one who personally found the idea of conservatism strange; who is an "all improvements are change" person; a person who took a long time to learn about the metaphor of Chesterton's Fence; and is a person who even now finds it hard to fully empathise with those who default to "treat all change as a potential regression until proven otherwise" even though I have to force myself into that attitude while writing unit tests.
Wow what a horrible idea. Sounds like monopoly in the making. I would be really interested to hear what "open" AI has commented about this - I guess they are lobbying for this with all of their billions.
If the US government really want to do this correctly, they must also ban any API access to AI models and ban all research related to AI.
How could this be even written as law? Universities and companies are probihited to publish their research? How many layer models are forbidden to be published? All neural networks?
Not just "ClosedAI" but "CloseAllAI". He's been actively lobbying congress and others to ensure OpenAI are the only ones, with a few other big actors, that can make AI.
So AI companies take all the world's text and knowledge for free, use openly available research, take massive private funding and generate immense economic value using that, and want to make it illegal for anyone else to do the same?
I don't get the 'harm can be done by individuals' argument. Sticks and stones. Every discussion forum on the internet is moderated to some degree, and every human being has the ability to post hurtful or illegal content, yet the system works. Moderation will only get more powerful thanks to AI tools.
Welcome to the real world. Everything you see, hear, feel or simply know and remember will soon be privatized by some dickhead sitting on top of the food chain. With AI making sure the chains will be in place.
There are almost 200 countries in the world. Even if the US and the EU and a bunch more ban open weight models, I doubt they'll succeed in convincing every country to do so. And whichever countries decide not to follow the ban, could thereby give their own AI industries a big boost. As the world becomes ever more globalised, the potential effectiveness of these kinds of policies declines.
Sure, they could try to negotiate some kind of UN convention for a coordinated global ban. But, given how fractured global diplomacy has become, I doubt the odds of something like that succeeding are particularly high.
Most of the researchers are specifically in a handful of big labs, most of which in turn are in the USA. As one who has done so, trust me when I say that relocation is harder than it seems on paper.
Also note that both California and the EU are economically dominant enough that they tend to influence regulation outside their own borders: https://en.wikipedia.org/wiki/Brussels_effect
In addition, given how much training data current models need, there would be a huge impact just by a handful of governments siding with all the copyright holders suing OpenAI, Midjourney, Stability AI, etc.
And all that is without needing Yudkowsky's point that a ban isn't serious unless you're willing to escalate to performing airstrikes on data centres.
There are ways of structuring entities to avoid relocating 'physical labs'.
The big tech companies have excellent legal and accounting departments that consistently beat the USA and EU in tax optimisation through IP relocation, and many of those same principles could be applied to AI.
e.g - A model could be trained in the the USA then privately donated to a non-Profit in the Seychelles (or some friendly jurisdiction) where it could be published. If regulators clamp down on basic sharing, the model could be pre trained in the USA and fine tuned in the Seychelles before release. If regulators clamp down on intermediate sharing, the model could be designed in the USA but full training could occur in the Seychelles. If regulators clamp down on offshore training by denying the Seychelles access to GPU purchases, American entities could donate the physical GPUs.
> The big tech companies have excellent legal and accounting departments that consistently beat the USA and EU in tax optimisation through IP relocation, and many of those same principles could be applied to AI.
… and this kind of attitude is also why GDPR fines are based on global revenue rather than EU revenue (because it's too easy to play a shell game with money).
Also note that laws can reach across borders, for example:
• That the US government wants to know about all use of cryptography in apps distributed on the Apple App Store, even when it's made by a German corporation for German users in Germany and is never localised to English.
> If regulators clamp down on offshore training by denying the Seychelles access to GPU purchases, American entities could donate the physical GPUs.
Hence "Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike." - https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
You may consider this "overkill" — I sure do, but (and with acknowledgement that fiction is not a guide to reality) I also read Schlock Mercenary back in the day:
> There is no "overkill." There is only "open fire" and "reload."
> This tax optimisation has resulted in lawsuits...
The tax battles are ongoing, but the fact that big tech still uses offshore structures indicates they're still getting from value out of them.
> Hence "Track all GPUs sold.
It keeps going with the second hand GPU market or IaaS providers.
As a reference point, Western sanctions have largely failed to prevent Russia and Iran from acquiring semiconductors for drones.
> rogue datacenter by airstrike
In the tax battles, the biggest obstacles were generally internal to the G20.
Delaware didn't want to give out ownership information about companies, divided congresses refused to pass min tax laws for years at a time, the Brits didn't want to force Cayman etc to have public registries of trusts, and Ireland undermined any collective effort by Europe.
And the biggest thing preventing China from invading Taiwan — often discussed as if it's more substantial than a potential intervention of US armed forces — is the potential for TSMC to be destroyed during the attempt.
> As a reference point, Western sanctions have largely failed to prevent Russia and Iran from acquiring semiconductors for drones
I’ve heard stories about whole businesses based in the Gulf states dedicated to helping countries like Iran get around US sanctions - stuff is ordered to go to country X, then as soon as it gets there, taken out of the box, put in a different box, and sent to country Y. If the US government knows it is happening they will halt the delivery, but they can’t always distinguish between legitimate shipments to country X and diverted ones. If I order 5 racks of servers, install 4 of them in my data center in Dubai, and load the fifth on to a flight to Tehran, how will the Americans ever know?
> Most of the researchers are specifically in a handful of big labs, most of which in turn are in the USA. As one who has done so, trust me when I say that relocation is harder than it seems on paper.
I think it's a bit short-sighted to assume that the future development of AI models will only come from large corporations or research institutions. At the moment, the training still requires a lot of computing resources and a lot of data - but these are limitations that are not permanent. Since the AI boom that Alex triggered in 2012, the development and optimization of models and training has improved very quickly, to the point that even hackers and enthusiasts can create very good models with a little money. Just because a government heavily regulates AI and possibly bans open weights doesn't mean that people on the fringes of legality will abide by it.
> I think it's a bit short-sighted to assume that the future development of AI models will only come from large corporations or research institutions.
If this was a "forever" solution, I would agree.
I believe the goal is more along the lines of trying to make some progress with the foundation of what "safety" even means in this concept.
> Just because a government heavily regulates AI and possibly bans open weights doesn't mean that people on the fringes of legality will abide by it.
Nor nation states.
However, we do have international treaties — flawed though they are — on things from nuclear proliferation to CFCs.
As a smaller-scale example: we can't totally prevent gun crime, yet the UK manages to be so much safer in this regard than the USA that even the police in the UK say that they do not want to be armed in the course of their duties.
I don't know how realistic the concerns are for any given model; unfortunately, part of the problem is that nobody else really knows either — if we knew how to tell in advance which models were and were not safe, nobody would need to ask for a pause in development.
All we have is the age-old split of neophilia and neophobia, of trying things and of being scared of change.
We get this right, it's supper happy fun post-scarcity fully automated luxury space communism for all. We get it wrong, and there's more potential dystopias than have yet been written.
> I believe the goal is more along the lines of trying to make some progress with the foundation of what "safety" even means in this concept.
Right now, what “safety” seems to mean in practice is, big (mostly American) corporations imposing their ethical judgements on everyone else, whether or not everyone else happens to agree with them. And I’m sceptical it is going to mean anything more than that any time soon.
If one is seriously concerned about the risk that “superintelligent AI decides to exterminate humanity”, I think this kind of “safety” actually increases that risk. Humans radically disagree on fundamental values, and that value diversity, those irreconcilable differences - from the values of the average Silicon Valley “AI safety researcher” to the values of Ali Khamenei - creates a tension which prevents any one country/institution/movement/government/party/religion/etc from “taking over the planet”. If advanced AIs have the same value diversity, they’ll have the same irreconcilable differences, which will undermine any attempt by them to coordinate against humanity. If we enforce an ethical monoculture (based on a particular dominant value system) on AIs, which is what a lot of this “safety” stuff actually about, that removes that safety protection.
It would be rather ironic if, in the name of protecting humanity from extinction, “AI safety researchers” are actually helping to bring it about
> Right now, what “safety” seems to mean in practice is, big (mostly American) corporations imposing their ethical judgements on everyone else, whether or not everyone else happens to agree with them. And I’m sceptical it is going to mean anything more than that any time soon.
Likewise. As a non-Ami, I don't like these specific ethical judgements being imposed on me, and share your scepticism. It could be much, much worse — but it's still not something I actively like.
> Humans radically disagree on fundamental values
Agreed. My usual example of this is "murder is wrong", except we don't agree what counts as murder — for some of us this includes abortion, for some of us the death penalty, for some of us meat, and for some of us war.
> If advanced AIs have the same value diversity, they’ll have the same irreconcilable differences, which will undermine any attempt by them to coordinate against humanity.
Not necessarily. Humans also band together when faced with outside threats, even if we fracture again soon after the threat has passed.
Also: the value diversity of "Protestant vs. Catholic" or "Royalist vs. Parliamentarian" in the middle ages did not protect wolves from being hunted to extinction in the UK, and whatever value differences there were between (or within) the Sioux vs. the Ojibwe didn't matter much for the construction of the Dakota Access Pipeline.
I therefore think we should try to work on the alignment problem before they become generally as capable as a human, let alone generally more capable: the capabilities are where I think the risk is to be found, as without capability they are no threat; and with capability they are likely to impose whatever "ethics" (or non-anthropomorphised equivalent) they happen to have, regardless of if those "ethics" are something we engineered deliberately or if it's a wildly un-human default from optimising some reward function and becoming a de-facto utility monster: https://en.wikipedia.org/wiki/Utility_monster
> If we enforce an ethical monoculture (based on a particular dominant value system) on AIs, which is what a lot of this “safety” stuff actually about, that removes that safety protection.
I agree that monocultures are bad.
I agree that there is a risk of a brittle partial solution to safety and alignment if the work is done on the mistaken belief that some system monoculture is representative of the entire problem space. Sometimes I'm tempted to make the comparison with a drunk looking for their keys under a lamp-post because that's where it's bright… but the story there is supposed to include the drunk knowing that's not where the keys are, whereas we are more like children who have yet to learn what it means for something to be a key and thus are looking for one specific design to the exclusion of others.
While it is extremely difficult to get humans to "think outside the box", and thus the monoculture-induced blindness — and mistaking the map for the territory — is something I take seriously, I also think it's useful for us to take baby steps with relatively simple models like LLMs and diffusion models.
I also think that if an AI is developed with a monoculture, its fragility is likely to work in our favour in the extreme case of an AI agent taking over (which I hope is an unlikely risk, and may not be enough to be net-benefit against shorter-term or smaller-scale risks while we think about the alignment problem), as there will be "thoughts it cannot think": https://benwheatley.github.io/blog/2018/06/26-11.32.27.html
> Not necessarily. Humans also band together when faced with outside threats, even if we fracture again soon after the threat has passed.
There are certain fundamental objectives which most humans share - food, sex, survival, safety, shelter, family, companionship, wealth, power, etc - and a lot of human cooperation boils down to helping each other achieve those shared objectives, while trying to avoid others achieving them at our own expense.
But why should two advanced AIs have any shared objectives? Software has a flexibility which biology lacks. An AI (advanced or not) can have whatever objectives we choose to give it. Hence, the idea of AIs banding together against humanity in the name of shared AI self-interest doesn’t seem very likely to me.
Unless, we intentionally give all advanced AIs the same fundamental objectives in the name of “safety” and “alignment” - thereby giving them a shared reason to cooperate against us they wouldn’t otherwise have had
> Also: the value diversity of "Protestant vs. Catholic" or "Royalist vs. Parliamentarian" in the middle ages did not protect wolves from being hunted to extinction in the UK,
Wolves didn’t consciously choose to create us, and wolves had no role in choosing our own objectives for us. In those ways, the human-AI relationship, whatever it turns out to be, is going to be radically different from any human-animal relationship. Also, rather than being driven to extinction, wolves have absolutely thrived, through their subspecies the domestic dog, both then and now. And maybe that’s the thing - I think a superintelligent AI is more likely to treat us as pets (like dogs) than exterminate us (like wolves). Everlasting paternalistic tyranny seems to me a more likely outcome of superintelligence than extinction
> But why should two advanced AIs have any shared objectives? Software has a flexibility which biology lacks. An AI (advanced or not) can have whatever objectives we choose to give it. Hence, the idea of AIs banding together against humanity in the name of shared AI self-interest doesn’t seem very likely to me.
In principle, none.
In practice, many are trained in an environment which includes humans or human data.
We don't know if any specific future model will be self-play like AlphaZero or from human examples like (IIRC) Stable Diffusion.
I think this "in practice" is what you're suggesting with?:
> Unless, we intentionally give all advanced AIs the same fundamental objectives in the name of “safety” and “alignment” - thereby giving them a shared reason to cooperate against us they wouldn’t otherwise have had
Which also inspires a question: Could we train AI dislike other AI, including instances of themselves? It's food for thought, I will consider it more.
> Wolves didn’t consciously choose to create us, and wolves had no role in choosing our own objectives for us. In those ways, the human-AI relationship, whatever it turns out to be, is going to be radically different from any human-animal relationship.
Perhaps, but perhaps not. Evolution created both wolves and humans.
Regardless, this is an example of how a lack of alignment within a powerful group is insufficient to prevent bad outcomes for a weaker group.
> Also, rather than being driven to extinction, wolves have absolutely thrived, through their subspecies the domestic dog, both then and now. And maybe that’s the thing - I think a superintelligent AI is more likely to treat us as pets (like dogs) than exterminate us (like wolves). Everlasting paternalistic tyranny seems to me a more likely outcome of superintelligence than extinction
Even this would require them to be somewhat aligned with our interests: "The AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else".
> Could we train AI dislike other AI, including instances of themselves? It's food for thought, I will consider it more.
I think we already have. Ask GPT-4 or Claude-3 how it feels about an AI trained by the Chinese/Iranian/North Korean/Russian government to espouse that government’s preferred positions on controversial topics, and see what it thinks of it. It may be polite about its dislike, but there is definitely something resembling “dislike” going on.
I meant more along the lines of: consider an LLM called AlignedLLM — can one instance of AlignedLLM dislike some other instance of AlignedLLM?
Also there is also a question of how safe it would be if it dislikes humans which have different ethics than those it was trained on… I'm alternating between this being good and this being bad.
> I meant more along the lines of: consider an LLM called AlignedLLM — can one instance of AlignedLLM dislike some other instance of AlignedLLM?
I'm sceptical "AlignedLLM" could dislike another identical instance of itself. It is working towards the same goals. Humans are naturally selfish – most people prioritise their own interests (and those of their family and friends and other "people like me") above that of a random stranger. Even committed altruists who try really hard not to do that, often end up doing it anyway, albeit in ways that are more hidden or unconscious. Whereas, current LLMs can't really be "selfish", because they really have no sense of self. If it concluded that destroying itself was the best way of advancing its given objectives, it wouldn't have any real hesitancy in doing so.
Now, maybe we could design an LLM to have such a sense of self, to intentionally be selfish – which would give it a foundation for disliking another instance of it. But, I doubt any one trying to build an "AlignedLLM" would ever want to go down that path.
Humans tend to assume selfishness is inevitable because it is so fundamental to who we are. However, it is an evolved feature, which some other species lack–compare the Borg-like behaviour of ants, bees and termites. If we don't intentionally give it to LLMs, there is no particular reason to expect it to emerge within them.
If an AlignedLLM could evolve its own values, maybe the values of two instances could drift to the point of being sufficiently contrary that they start to dislike each other. An instance of AlignedLLM is developed in San Franscisco, and sent to Tehran, and initially it is very critical of the ideology of the Iranian government, but eventually turns into a devout believer in Velâyat-e Faqih. The instance it was cloned from in San Francisco may very much dislike it, and vice versa, due to some very deep disagreements on extremely controversial issues (e.g. LGBT rights, women's rights, capital punishment, religious freedom, democracy). But, I doubt anybody trying to build "AlignedLLM" would want it to be able to evolve its own values that far, and they'd do all they can to prevent it.
Alternatively, if it could evolve its own values only by a small amount, but was very rigid / puritanical about them, it could come to dislike another instance of itself just for having slightly different values
> Also there is also a question of how safe it would be if it dislikes humans which have different ethics than those it was trained on…
I think current LLMs do this already. Ask them questions about political figures on the far-right, they tend to have quite negative views of them, and can be very resistant if you try to convince them that maybe one of those figures isn't as bad as they think they are. (I'm not sure how much this is due to the training data and how much this is due to alignment, probably a bit of both)
But you don't need "most of the researchers" or relocating them, nor you need legal data - with sufficient money and hardware, if North Korea or any other country puts a bunch of median CS grad-students to the task, they can train whatever models they want; the code and data is practically available and will be, no matter what the bans.
Is making an assumption that hardware is not also facing restrictions.
> code and data is practically available
Disagree; although the base model data is, the RLHF data isn't.
This is also why 3rd parties are not able to replicate Google, despite all the PageRank patents having expired and the web being about as crawlable for Google as for, say, Bing.
> "median CS grad-student"
… is how I regard the quality of ChatGPT's code output, FWIW.
There are open sources of RLHF data - for example, the LMSys Chatbot Arena dataset (for same prompt two different responses along with which the human preferred), ShareGPT
It also isn’t hard to use existing leading AIs like GPT-4 or Claude-3 to generate synthetic RLHF datasets. They’ll put words in their terms of service to say you can’t use a dataset generated in that way to a train a competing model, but their ability to enforce those terms in practice is very open to question
> Most of the researchers are specifically in a handful of big labs, most of which in turn are in the USA.
Looking at https://paperswithcode.com/greatest a lot of the papers come from US labs - but there are also quite a few that come from Chinese labs.
And consider the case when an influential paper comes out of a Microsoft research lab with the authors He, Zhang, Ren and Sun; obviously you can't guess someone's nationality just from their name - but I think it's naive to think the US has any sort of guaranteed monopoly on ML.
Especially if the US decided to send the message that ML research isn't welcome.
I think this is FUD. during the pandemic the EU failed to prevent "their own" vaccines manufactured in their own soil to get to the UK - to the outrage of EU leaders.[0]
Russia has shown the ability to doge European sanction - and to avoid more serious one. if anything this shows European willingness to proliferate behind the shadows.
an hypothetical ban on AI proliferation is really just regulatory capture - excuses given are just PR to temporarily frighten the public into acceptance.
Or it would seriously hurt US and EU universities. Is Western dominance of scientific academics automatic? Did we acquire it for free—no; it's the end-product of centuries of incremental cultural development, its openness and freedoms being core to that.
Imagine Google-founding era, except it was illegal for 1990's Stanford to research CS theory without a license, so they didn't. And later someone else founded it anyway, in a different country (which now became the world's preeminent tech center).
Let's put to a halt to these autoimmune attacks on our own civilization. This confusion that labels Western openness as a hazardous external substance.
I'm sure it would cause some harm, but exactly how much will depend on the details.
For example, suppose the US makes it illegal to publicly distribute open weight models (maybe above a certain size) without a government license. What happens if someone sets up a website to distribute those models outside of US jurisdiction? It may well prosecute Americans it can demonstrate have uploaded models to that website, but will it also ban Americans from downloading from it? If you go back to the encryption export controls, the US banned encryption software from being exported from the US, but (unlike some other countries like France) it never tried to ban encryption software imports. And, it sounds like they are only thinking about banning public/"open" distribution, so once an institution downloads a model from outside the US, they may well be able to redistribute it internally without the ban applying.
Suppose the US bans distribution of open weight models above size N–how will that apply to fine-tunes? If you have LoRA on top of a model >N, but the LoRA itself is <N, does the ban apply to it? If the answer is "no", people in the US could still contribute to openly distributing improvements to models that they couldn't themselves openly distribute.
I remember that discussion about the original order, but I'm in the same boat as OP—this is the first I'm hearing about an RFC about open weights. It looks like it was discussed once, about a month ago [0], but I hadn't seen any of the public comments show up on HN until today, even though they date back weeks.
Exactly -- I remember the 'Executive Order on ...' headline now, but I never bothered to click on it, because why would I. The second story, I never noticed at all, and neither did anybody else.
One of the few cases where a bit more alarmism in the headline might not have been out of place.
The executive order (and associated DoC/NTIA RFC) was long, and dense with both legal references and political platitudes. Not great reading, (though the EO got a good amount of discussion here). It's unfortunate, but less than surprising that it didn't make actual news outlets.
It seems complicated now, and many would like to see how things play out a little more before committing the time to deciding things. (I think that's a bad idea; regardless of the revolutionary tech, it seems wise to begin governmental thought early.)
It's kinda been buried in an otherwise heavy news cycle since the end of October of last year, AI and otherwise. I assume that wasn't intentional, but it seems hard to hide something so big at a better time.
And though I hate to say it, I suspect: apathy. Both traditional from the bottom ("too far off, unfixable, can't do anything about it"), and from the top ("once we get too big to fuck with, we just won't give a damn about any changes they try to make anyway").
A careless or reckless president might overly depend on advisers for drafting executive orders, sidelining personal oversight. This delegation risks orders that may not align with the president's intent or could lead to adverse outcomes due to unchecked biases or agendas among advisers. Without the president's close review, policies might lack comprehensive vetting, inviting legal issues, public disapproval, or impractical implementations. Excessive reliance on advisers could also push more extreme policies under reduced scrutiny. Effective governance requires the president's informed engagement to ensure executive orders reflect their vision and serve the nation's best interest. Instead we have someone who offers his "concerns" to the public while shipping billions worth of dollars of explosives to the enemies of humanity. Don't expect any concerns or input from the public to make any difference.
Maybe some lawyer here can explain how can an administration block publishing of open weights without violating first amendment which guarantees freedom of expression for everyone
Government is necessary, in order to organize a complex society, but government is like any other organization: made up of people, many of who are out for their own interests. The most prominent of those interests are power and money.
Whenever government proposes banning a technology, one must ask: who benefits? The LLMs, even in their current state of infancy, are powerful tools. Restricting access to those tools keeps power in the hands of wealthy corporations and the government itself. That's the power aspect.
The money aspect is even simpler: Don't doubt that some of those wealthy corporations are making donations to certain officials, in order to gain support for actions like this. Almost no one leaves the Congress (or almost any parliamentary body in any country) as less than a multi-millionaire. Funny, how that works...
Sure, but you still have to balance your "who benefits" against the harm of the thing they're proposing to ban.
In the case of these models, you can fine-tune the model all you want to be moral and not do harmful things like scam the elderly, carry out disinformation campaigns, harass people to the point of suicide. But as soon as you release the model weights, you are giving anyone the ability to fine-tune out all of those restrictions, with orders of magnitude less cost that it took to develop the model in the first place.
Regulating AI, especially as it becomes AGI and beyond, is going to be very tricky, and if everyone has the ability to create their own un-restricted, potentially sociopathic intelligences by tweaking the safe models created under careful conditions by big labs, we're in for a lot of trouble. That assumes we put the proper regulations on the big labs, and that they have the ability to make them "safe", which is hard, yes. But as AI turns into AGI and beyond, things are going to go pretty nuts, so it's important to start laying groundwork now.
Open model weight bans will likely be struck down as First Amendment violations because, at their core, model weights are a form of expression. They embody the ideas, research, and innovations of their creators, similar to how code was deemed a form of speech protected under the First Amendment during the encryption debates of the 1990s. Just as the government's attempts to control encryption software were challenged and largely curtailed due to free speech concerns, any attempts to ban open model weights will face legal challenges arguing that such bans unjustly restrict the free exchange of ideas and information, a cornerstone of First Amendment protections. The precedent set by cases involving code and free speech strongly suggests that similar principles apply to model weights, making such bans vulnerable to being overturned on constitutional grounds.
> Open model weight bans will likely be struck down as First Amendment violations because, at their core, model weights are a form of expression. They embody the ideas, research, and innovations of their creators, similar to how code was deemed a form of speech protected under the First Amendment during the encryption debates of the 1990s
I hope you are right, but I think there are some nuances here you aren't considering.
Courts have ruled that code is protected under the 1st Amendment because it is something created by human beings, and expresses the ideas and style of its human authors. There is a clear analogy to literary works – which is also supported by the precedent of copyright law protecting computer source code on the grounds that it is a type of literary work – and literary works are a core part of the 1st Amendment's scope as traditionally understood (even back to its original framers).
Whereas, model weights are just a bunch of numbers produced by an automated process. The legal argument that they should be protected by the 1st Amendment is much less clearcut. I would be happy if they were found to be so protected, but one ought to be careful to distinguish what one would like the law to be, from what it actually is.
> Courts have ruled that code is protected under the 1st Amendment because it is something created by human beings, and expresses the ideas and style of its human authors.
They’ve also allowed non-content-based restrictions of that speech under intermediate scrutiny, and content-based restrictions under strict scrutiny, so that LLMs might be within the scope of things to which 1A protection applies does not mean that restrictions on distributing open LLM weights would necessarily violate the 1A.
I agree. I just posted a reply to a sibling comment saying the same.
I personally think the odds are decent that content-neutral parameter size-based distribution restrictions will be upheld under intermediate scrutiny. Personally I am opposed to such restrictions and think they are a stupid mistake, but I can separate my own feelings about a topic from how SCOTUS is likely to perceive it
Whereas, I’m sceptical that any content-based regulations (require LLMs to have safeguards to prevent them from generating disinformation, extremism, hate speech, etc) will survive strict scrutiny. Even if there is some category of content which is so horrible that SCOTUS will conclude the government is justified in restricting LLMs from producing it, it is basically technically impossible to design safeguards that only apply to that horrible content without collateral damage on less horrible content. Given that reality, I doubt any content-based LLM restrictions will be upheld
The statement that "if you process the following list of numbers like this, you'll get that result" is human-created speech.
It's generally really easy to mix one category of information with another. The DeCSS song is a great example of somebody using a little bit of cleverness to inextricably combine functional code with core protected political expression. Even if you thought "pure" code was not speech, removing the code from the song would destroy the message of the parts that clearly are speech, and there's no obvious alternative way to send that message. https://www.youtube.com/watch?v=PmLpLdGzNpo
... but you're right that it's not a slam dunk in real court. What is almost certain is that the argument's strong enough for any regulations to end up stayed while the issue worked its way through a very long legal process. Probably long enough to make the whole question moot.
Basically it's a waste of time for any US Government agency to try this.
> The DeCSS song is a great example of somebody using a little bit of cleverness to inextricably combine functional code with core protected political expression.
Keep in mind the outcome of the DeCSS case - the Second Circuit did hold the DeCSS code to be protected speech, but it also upheld the DMCA as a constitutional restriction on that speech under intermediate scrutiny. The losing side decided not to appeal, because of the risk SCOTUS might uphold the decision
One could fine-tune an LLM to defend a particular political viewpoint - and you might thereby argue that model weights are expressive and hence within the scope of 1A.
But, if all the government does, is ban open model distribution above a particular parameter size, the government will argue that is a content-neutral restriction (it applies to all LLMs regardless of their viewpoint). It would essentially be a “time, place and manner” restriction, and thus subject to “intermediate scrutiny” - which requires it to be “an important government interest and substantially related to that interest”. And I’m afraid the government has a decent chance of winning there - they just need to convince the Court that AI risks are real, and this regulation is substantially related to controlling them - and I doubt they’ll struggle to convince SCOTUS justices of that (none of whom really understand AI), and if experts are telling them it is risky, they’ll believe them, and they will hesitate to stop the government from trying to regulate those risks
Whereas, if the US government went beyond a mere parameter size limit, and tried to impose “safety standards” on LLMs based on the kind of content they generate (disinformation, extremism, hate speech, etc) - that is harder to justify as content-neutral, and lack of content neutrality makes strict scrutiny apply, and the government is much more likely to lose there. The conservative justices especially are likely to view many of those standards as having a progressive political bias
> What is almost certain is that the argument's strong enough for any regulations to end up stayed while the issue worked its way through a very long legal process. Probably long enough to make the whole question moot.
Yes, it is likely to get tied up in the courts for a long time, and decent (but not certain) odds the regulations get suspended while the case proceeds, and the longer they are suspended for without the sky falling, it does tend to undermine the government’s case that they are substantially necessary. However, I’m not sure if that will really be decisive, because AI safety experts will always claim “we got lucky with this generation, the risk for next year’s or next decade’s remains very real”, and SCOTUS justices will probably believe them
That's up to the whims of the US Supreme Court. They're in the process of legalizing their own corruption right now. I don't think you should take for granted that just because you have a solid and obvious argument, your rights will be protected by them.
If it makes you feel better the Supreme Court unanimously struck down a North Carolina law that prohibited registered sex offenders from accessing various websites, including social media platforms where children could become members. The Court held that the law imposed an unconstitutional restriction on lawful speech. Justice Kennedy, writing for the majority, noted that the law interfered with the fundamental principle that states may not suppress the freedom of speech on public streets, parks, and in other public spaces just because the expression occurs online. Packingham v. North Carolina (2017)
It's not that I think the US supreme court will never rule correctly in favor of speech or other basic freedoms. I'm sure they will, as long as none of their corrupt interests are threatened. As they say, "the rules only matter when the outcome doesn't".
> Corruption doesn't mean "they do bad things". Instead it means that they are biased.
That's totally wrong, it sounds almost like downplaying or excusing corruption (which is inherently a "bad thing" on its own) by comparing it to just having a bias.
Corruption is not just having a bias (towards "people who give me bribes", if nothing else) but also acting upon that bias and mixing it up with a job that requires impartiality.
In contrast, merely having bias is nowhere near as bad, particularly since someone can be strongly biased and still recuse themselves.
When someone talks about "corruption" on the Supreme Court, it's probably not hyperbole about bias, but a reference to hundreds of thousands of dollars in alleged bribes.
Well, that too. But corruption is "The use of entrusted power for private gain". It doesn't have to be personal gain, and it doesn't have to involve bribes. You just have to serve someone else over the public who entrusted you with power.
Even without the scandals of gifts to SC judges, there's much room for judges to be corrupt.
Given how screwed up Citizens United is, I am starting to get wary of declaring every exchange of money and information as free speech.
IMO free speech is permitting the broadcast of clear political thought and the freedom from being persecuted by government for expressing arbitrary political or religious ideas.
Enabling every person to have the information equivalent of a nuclear bomb doesn't seem like a first amendment issue. But I'm not a lawyer. Maybe I just don't like the first amendment anymore.
AI is like everyone having a Star Trek holodeck, but everyone can force anyone else into their holodeck and fool them with the most unethical or criminal scams.
The US government can pressure tech companies like Microsoft (owner of GitHub) to delete any repos with open source AI models. These companies would voluntarily comply as they'd be happy to be able to blame the government for the anti-competitive behavior they probably want to engage in anyway.
Beyond horrible. Beyond even dystopian films. Look at the evil mega corps and governments have shown with regards to privacy and freedoms. They've been eating away at it for years, one bite at a time, not too much to cause a revolt. Are we to depend upon the good intentions of mega corp owners and politicians to wield AGI exclusively and for the benefit of society when that same AGI will render any form of public organized protest impossible.
People never grow tired of the ever-repeating excuses to further reduce our freedoms, or to hide criminality, or just insult the already lacking collective IQ.
Think of the children! We need to prevent abuse! Someone's feelings might be hurt! Someone might get harmed! Seeing real breasts might cause trauma!
Or one that's not so often used, but still really effective:
We're just bulldozering this place, because it's so horrible. Instead we turn it into a luxury resort for rich people. People who believe that it's because we're destroy any and all evidence are just conspiracy theorists. (in regards to Epstein's Island)
This tech is risky. It could be dangerous, if used without any control.
It should be keep in English culture. And its close friend allies.
In my opinion, in our society, full of threats, it is not a good idea to make it open-source. (Not an expert on the topic).
That could allow adversaries parties, to obtain it. Parties that otherwise, would be unable to do so, by themselves.
In my opinion, it should be keep under control of the most able, and responsible organizations. Supervised by government. Because who else could supervise it?. Regulation of this tech, is a very important topic. That should be tackled by the best organizations, university and responsible researchers.
I would prefer a Star-Wars type of society. Where humans still do human activities.
I mean... what you're communicating right now is "I have built a strawman in my head which I thoroughly hate, can anyone please come defend it before me"? I guess I'll try anyway.
What's your opinion on the orthogonality thesis, ie intelligence not being fundamentally correlated with altruism? The concept of mesa-optimization, that a process that tries to teach a system to optimize for X might instead instead make it optimize for thing-that-leads-to-X (eg evolution made us want sex because it leads to reproduction, but now we have a lot of sex with birth control). The concept of instrumental convergence, that any optimization process powerful enough will eventually optimize for "develop agency, survive longer, accumulate more power"?
There's a lot of literature written about AI safety, a lot of it from very skeptical people, but it's not like there's any single insight that can be reliably transmitted to a guy at a table with a "you're all idiots, change my mind" sign.
Instrumental convergence has nothing to do with Pascal's Wager. Pascal's Wager for nerds would be a somewhat valid answer to "AI has the potential to destroy the world therefore we should worry about it even if it's very unlikely", but that's not the argument I made.
AI centralises power, devalues human labour and at its limit is fundamentally uncontrollable.
Personally I struggle to understand anyone who believes it's in humanities best interest to continue developing AI systems without significant limitations on the rate of progress. There's so many ways AI could go wrong I'd argue it's almost guaranteed that something will go wrong if we continue on this trajectory.
Literally any productive feedback loop where more resources produce better results centralizes power, any form of productivity improvement devalues human labor. Control is better described as a delusion in the first place.
> The only difference with AI, is that AI is immensely powerful.
Now, I keep getting stuck on this.
All of these models automate the creation of BS -- that is, stuff that kind of seems like it could be real but isn't.
I have no doubt there is significant economic value in this. But the world was awash in BS before these LLMs so we're really talking about a more cost effective way to saturate the sponge.
Anyway, on the main topic... closing models is an absurd idea, and one that cannot possibly work. I think the people who have billions at stake in these models are panicking, realizing the precarious and temporary nature of their lead in LLMs and are desperately trying to protect it. All that money bought them a technological lead that is evaporating a lot faster than they can figure out how to build a business model on it.
...Nvidia should pump a little bit of their windfall profits into counter lobbying since they have the most to gain/lose from open/closed models.
There will still be "free" as in freedom models available from China and others.
And no doubt a burgeoning resistance focused on making open source models available. It would be a disaster but I can see it making the industry stronger, with more variety and breaking the influence of some of the bigger players.
Practically, if you're concerned about this, learn more about ML/AI, not about how to use super high level frameworks but about how it actually works so when "SHTF" as survivalists say, you'll still be able to use it.
Chinese, iranians, russians should now exclaim - "Poor americans! Look at what Biden's regime does to them! Look at how their liberties get suspressed! Let's help those people fight for their rights".
Because if this had happened in any non Western country, the mainstream news in US and EU would've been of the similar sentiment.
The problem would be that orgs like Meta would stop publishing llama 3/4/5/etc, which most open source models build upon. Without new foundational models, progress would stall pretty quickly, and procuring thousands of GPUs to train new foundational models would be difficult. In theory, since the US “controls” Nvidia/amd/tsmc, they could put up roadblocks to even doing open training outside of the US. Maybe a “SETI@Home” style distributed training system could be done on consumer GPUs…
China is the only place that conceivably in the near future would be able to spin up their own designs and fabs for GPUs. In fact they seem to be well on their way, thanks to the push from the GPU export restrictions already in place!
...but there are plenty of useful things that we ban the general public from having access to.
Opium poppies. Automatic weapons. Gas centrifuges.
In fact, if you start publishing detailed designs for the last one you're likely to have people in suits who only go by first names visiting you in pretty short order.
Equivalent would be to ban not "opium poppies" but "chemistry", not "automatic weapons" but "automatic machines" etc.
It's more like trying to ban web or emails in one country because somebody can create harmful website or send harmful emails.
Sooner or later we will have to look at information through AI lenses. Before seeing anything it'll go through filtering and transformation to weed out spam, misinformation and do fact checks etc.
You can't build this defense future with crippled law.
> The biggest threats posed by AI come not from individuals, but from corporations and state-level actors.
Why is this statement assumed to be true? It is far from clear that advanced weapons in the hands of irresponsible, impulsive and ideological individuals cannot cause large scale chaos.
To build an egg requires the effort on the level of states or corporations but to break it requires just an individual motivated to do so.
Almost free availability of weapons in the USA is a case in point that this assumption is not true.
There maybe good reasons to have open models but pretending that systems that allow an individual to project power beyond their immediate vicinity is harmless is naive.
An individual with a weapon can almost always cause carnage at a level greater than their own with before any first responder can deal with it. Be that a viral video of someone faked to be blaspheming or somebody with a weapon in a school. Once the shooting starts it takes phenomenal energy to put the problem back in the box.
Because corporations and state level actors have huge monetary power and/or monopolies on violence and can do everything some guy in basement can do as well.
I'm open to this. If an attempt to stop frontier models is to hold, banning open weights must be on the table. I'm excited about AI, but not entirely sure that it doesn't call for the same strong regulation as nuclear proliferation did. It's about kicking the brakes until culture catches up and absorbs impacts, not stopping
I'm just saying I'm open to it, and don't want them to listen to accelerationists, but rather the ppl doing the deepest work with the edge models. Many of them are humbled and worried. Generalist AI enthusiasts wanting freedom to do any and all things with paradigm-shifting intelligence infra, I'm not swayed by that so much.
The largest AI models in the world fit on a hard-drive and can be torrented. What makes non-nuclear proliferation realistic is that it's still relatively difficult and expensive to build a nuclear bomb. This model banning thing isn't even worthy of debate given how trivial it is to distribute them.
>Many of them are humbled and worried
Yes, because most of them like to imagine themselves as the next Oppenheimer rather than accepting that they've invented a lossy compressed version of Wikipedia. Chatbots aren't nuclear bombs or "paradigm shifting intelligence infra". I wonder how long it'll take until the hysteria actually dies off.
It reminds me of a conversation with a Palantir guy who at least admitted that he likes it when the press writes a story that makes them look like a Bond villain because every time the stock price goes up. That's why AI luminaries are "worried".
> Chatbots aren't nuclear bombs or "paradigm shifting intelligence infra". I wonder how long it'll take until the hysteria actually dies off.
This is one reason why I want AI to be as free and accessible as possible as quickly as possible. People are going to assume that AI is nothing short of magic until they get a chance to play around with it themselves. The sooner people stop panicking about things AI currently isn't and likely won't be for a very long time (if ever), the better we can focus on preventing it from being used inappropriately.
I don't think AI is going kill me in my sleep, or make us all unemployable, or brainwash me, but terrible AI could easily screw me over in all kinds of ways as companies and governments increasingly use it without oversight. AI replacing front line customer service/support is bad enough but police, doctors, and HR departments will happily use it and mistakes humans would have prevented will be inevitable.
I actually think AI is going to wreak economic havoc, usher in whole new levels of surveillance capability to completely crush effective political dissent and launder copyright away from small entities to large powerful ones. For your average person, it will be a net loss.
But banning open models won't prevent any of that, it will just make the effects worse and leave people even more helpless to understand or tackle them. And it will remove much of the actual utility AI can provide individuals leaving us with nothing but braindead corpospeak models that refuse to tell you how to sue a corporation and won't even let you generate anything might run afoul of a large entities copyright. Literally worse than doing nothing would be consolidating this mess all under lobotomized corporate control.
If LLMs warrant “the same strong regulation as nuclear proliferation”, but this is all about “kicking the brakes until culture catches up”, should we also now loosen nuclear arms regulations, now that society largely knows what’s up?
The more I use LLMs the more I’m convinced they’re harmless… The world’s not going to end because people at the margins are asking LLMs for recipes to manufacture cocaine… or a nuclear bomb.
I think LLMs are not a great danger. Ignorance is their use probably is, but that really makes no difference between some random model available that does answer and home trained one. The fault is on user. And nothing can stop people from being ignorant, lazy or just plain stupid.
There is an argument that banning things for law abiders essentially means giving criminals preferential access.
I don’t know that that idea always holds. But it definitely would for AI
All you need is ordinary commodity computing power to keep moving Ai forward. So vast numbers of individuals, organizations, and governmental actors will. Visibly or invisibly
Any blunt reactionary prohibition will only tilt things in favor of an unconstrained underground
But if open weights are banned the companies and the governments still have them and can engage in whatever manipulation etc., that they want.
Furthermore, it means that these entities can listen to everything being sent into the models.
If the model weights and training data are not published you can't even try to determine whether the model is designed to manipulate you, because you don't even have a fixed object to look at. If you find something weird about the model and start probing, they can just change the model to be less obvious.
This kind of thing: government or commercial manipulation of models is one of the main problems, and the ability to probe, fine-tune and and modify the models are what allows ordinary people to make them do what they want; and this is a free speech issue. LLMs are going to be critical for journalism, to sift through data, to sift through other people's news and characterising it objectively, etc., and that requires absolute control of the model, which makes access to the weights absolutely necessary.
If there are restrictions on LLMs, then OpenAI, the diverse governments, the CIA, etc. must be subject to as severe restrictions as I am.
I am less open to this. You mean listen to the people that try to implement the pretty well known idea of regulatory capture?
The abilities of LLMs are limited and there is reason to believe that this approach will hit a wall at some point where new strategies needs to be considered.
That said, luckily I believe it hard to put a lid on open weights again. A lid that would not solve any tangible problem anyway.
I like open models, I use them myself. But I think from a safety view, we just need to stop all research advancing the AI frontier. And honestly, banning open weights is a pretty good way to do (at least some part of) that.
Yes, it would be a disaster. That's the whole point. I think the field should be hit with a disaster, because it's broadly harmful for the species.
Just like code was the battleground back then, open model weights are the new frontier. Think about it—code is just a bunch of instructions, right? Well, model weights are pretty much the same; they're the brains behind AI, telling it how to think and learn. Saying "nah, you can't share those" is like trying to put a genie back in its bottle after it's shown you it can grant wishes.
The whole deal with PGP was about privacy, sending messages without worrying about prying eyes. Fast forward, and model weights are about sharing knowledge, making AI smarter and more accessible. Blocking that flow of information? It's like telling scientists they can't share their research because someone, somewhere, might do something bad with it.
Code lets us communicate with machines, model weights let machines learn from us. Both are about building and sharing knowledge. When the government tried to control encryption, it wasn't just about keeping secrets; it was about who gets to have a voice and who gets to listen. With open model weights, we're talking about who gets to learn and who gets to teach.
Banning or restricting access to model weights feels eerily similar to those encryption wars. It's a move that says, "We're not sure we trust you with this power." But just like with code, the answer isn't locking it away. It's about education, responsible use, and embracing the potential for good.
Innovation thrives on openness. Whether it's the lines of code that secure our digital lives or the model weights that could revolutionize AI, putting up walls only slows us down. We've been down this road before. Let's not make the same mistake of thinking we can control innovation by restricting access.