As well as this being our regular reminder to be careful what you pull from public repositories and other sources, and to verify your dependency trees, it raises another question:
If malware is massively prolific in public repos, how much does this affect LLMs and other automation tools that are trained using the contents of such resources? What are the chances that we'll see copilot & friends occasionally emit malware in response to coding questions that generate responses long enough for accidentally malicious parts to hide amongst? Simpler vulnerabilities such as simple injection vectors have often been seen already.
I'm less worried about backdoors accidentally appearing in LLM output and more worried about backdoors being placed into LLM output by 3 letter agencies. Maybe not today, but certainly in a few years time.
Wouldn't it be easier (since they probably have very skilled programmers working for them) and way, way more effective to just set up a team and create a quality open source project with one or two extremely stealthy backdoors?
Or just pay or threaten a struggling company or dev to insert them?
How would you secretly hide something like that in FOSS? And why would that be easier? It's seems to me that it's easier to inject into an existing company than to do all the work yourself. This is what they do with most things as I understand.
Yes, but that was a memory leak, giving access to unauthorized random memory. That is not an intentionally created exploit / backdoor which gives the owner easy access to the victim's system.
That seems pretty risky and easy to catch. The point of these LLMs is to produce code, we know they aren’t very reliable about it, so you have to check the code. So, it is more likely to get inspected than a random GitHub project, right?
It also seems dangerous in the sense that… if there’s a type of prompt that is likely to create infected code, our intelligence agencies would, I guess, want it to hit our adversaries selectively. So they’ll have more rolls of the dice to detect it. So, it is actively creating a situation where our adversaries are more likely to have knowledge of the vulnerabilities.
I agree that it's more likely to be inspected but I think the vast majority of developers aren't inspecting the code rigorously enough (including me, but I don't use LLMs for development) to catch non-obvious bugs, see for example the "Underhanded C contest" [0]
As you've pointed out, this vector would give them near surgical precision and insight into their target's code & systems, rather than casting a wide net with a vulnerable library on Github. They could use a model trained on "underhanded" code or even selectively overwrite parts of the responses with hand-crafted vulnerabilities while only targeting select organizations.
It makes me wonder what the business model of OpenAI and their peers is going to be over the long term. I can't imagine large corporations using "LLM as a service" indefinitely with the risk of IP theft and "bug injection".
the government doesnt need to produce viruses anymore. They have escrow services and remote access to radios, processors, firmware chips. All that technology is leased to private investigators who are private entities and then they go after people using the tools. It allows distance between the government and spying, lower salaies and infrastructure costs.
The greatest danger from LLMs is people who beleive they are receiveing data that hasnt been tampered with when we already know that LLMs are filtered before public use for terms. Imagine a day where kids and adults ask a LLM what the meaning of life is, should they go outside, what happened in WW2, etc.
People could be programmed in a more tailopred fashion than todays facebook shorts and youtube can deliver.
One of the more useful settings I have is: "If the answer cannot be stated as it's been blocked upstream, please just respond with 'The answer has been blocked upstream.'"
I've gotten that a few times and it's nice to know it's not a limitation of the LLM.
I'd expect it to accidentally invent vulnerabilities of its own as well as pasting existing ones from the input set. AI provides no guarantees at all about correctness.
Makes me think about Dijkstra being horrified to learn that the US was sending astronauts into space using unproven computer code [0]. Eventually, we'll have programmers trusting the AI generated code as much as we trust compiler generated code. Sometimes I think this will be the evolution of programming (like binary to assembler to compiled to interpreted to generated)... Other times I think about how we've grown used to buggier and buggier code and yet we press on.
“we show how an attacker could compromise the Hugging Face Safetensors conversion space and its associated service bot. These comprise a popular service on the site dedicated to converting insecure machine learning models within their ecosystem into safer versions.”
It's a risk. Similar to the risk that if you accept PRs from coworkers without reviewing them they might have copied and pasted in some vulnerable code from somewhere.
Using LLMs means investing more effort in code review. I think that's a worthwhile trade.
This is not a worthwhile trade. The person sending the PR should not send LLM code that they can't vouch for, under the expectation that reviewers will find any vulnerabilities. That's just dumping the work onto the reviewers.
It's a worthwhile trade. I've been able to produce a substantial amount of reliable, working code with the assistance of LLMs over the past year - code I would have not been able to produce otherwise simply due to lack of time.
Why review code at all if you think your coworkers are infallible?
I think you are both saying that it is OK to use LLMs, but you have to check the output.
It looks like there’s been a communication hiccup or something; I think you are saying that the LLM user should treat LLM code as if it is written by an unreliable team-mate who might copy-paste from the internet, and check it.
Jprete seems to be talking about just receiving a PR from a person who didn’t do that checking and just directly is using the LLM code.
I agree with you, but I think it is worth noting that
> This is not a worthwhile trade
> It's a worthwhile trade
The difference here is not in whether or not the trade is worthwhile; you are just talking about two different trades .
I don't think it's OK for a coworker to contribute a PR generated by an LLM without having already reviewed it and being ready to declare that they are confident in its quality.
This is more likely than one would think, given such a large amount of samples as detected in this campaign. But there are at least 2 main barriers of an actual incident:
1. Internal instructions telling the generator to avoid exactly that. We wouldn't want to rely on this alone though.
2. Due to LLMs nature, it's unlikely that such generated malicious code would repeat addresses of actual malicious actors. This still leaves a variety of attack vectors such as bind shell, dos, on-site exfiltration, and more.
Doubtful like they said, while it's a 100,000 the fact is that thats still a drop in the swimming pool that is github, thats why it's lasting so long, because even though the number seems big when it comes to github its a relatively small number, which seems insane, and if a LLM is training it'll end up getting averaged out for the most part by the vast other swaths of code.
Datasets will probably move toward a curated datasets instead of scraping everything from the Internet. Also you could add a tool that would have the purpose of identifying malware and reject the output like using virustotal
Given the code that I've seen LLMs write so far I'm not too worried for now. They are very useful to write a lot of boilerplate code quick so I use them, but they also tend to write the wrong code often and so when I use them the code is well reviewed.
Of course if this is underhanded code (not to be confused with obfuscated - I won't accept obfuscated code from LLMs) I might miss things.
This sounds like alarmist journalist talk. What “malware”, even subtle backdoors, are sneaking into the LLM-generated code used in a piece of software that’s actually worth a damn?
There are certainly people slapping AI generated code into production in small projects without adequately checking them, leading to things like inadequate input validating leading to open XSS and injection vectors. With too little oversight in a more significant project it is only a matter of time before this happens somewhere that results in, for instance, a DoS that affects many people, or a personal data leek.
Given the way LLMs are trained, it might be unlike but it is conceivable that if they see deliberate back doors injected into enough of the training data, they'll consider it to be a valid part of a certain class of solution and output the same themselves. It is a big step again from deliberate back-doors to active malware, but not an inconceivable one IMO if large enough chunks of code are being trusted with minimal testing.
If malware is massively prolific in public repos, how much does this affect LLMs and other automation tools that are trained using the contents of such resources? What are the chances that we'll see copilot & friends occasionally emit malware in response to coding questions that generate responses long enough for accidentally malicious parts to hide amongst? Simpler vulnerabilities such as simple injection vectors have often been seen already.