Aside from obviously malicious bulk spamming like this, I'm seeing project maintainers get increasingly exasperated at users asking for help to fix some nonsense code which they eventually admit was generated by ChatGPT after some prodding. One Discord server I'm in has already instituted a "thou shalt not use ChatGPT" rule because it's wasting so much of the regulars time on deciphering confidently-wrong AI gibberish that someone took at face value.
Whoever said that generative AI was going to raise the internets noise floor was right on the money.
Friendly PSA: If you're going to use ChatGPT to generate code, do everyone a favor and add something like "...and write a passing unit test" to the prompt.
If you want to really do everyone a favor, consider that TDD has never been easier. Don't write a prompt, write a unit test-- and instruct GPT to write the code that makes it pass.
(Don't ask AI to do your work...ask it to finish your work.)
ChatGPT tends to over mock the module to the point it will always pass the test, so except for utilities module, this isn't helping much either. This is especially common in JS environment.
If you think about it, it does make sense why: the prompt tends to have no where near the context of how the module should be delineated. Normally the programmer knows how the abstraction within his program is separated, but we have no good way to tell it to chatGPT. That means unless your program structure is still following one of the common structure and hasn't morphed into your business domain structure, ChatGPT will mock the wrong layer/ abstraction in the test suite.
I don't know JS, but it works pretty well for me in Python. Where it goes astray, I'll put comments indicating what it's supposed to do and GPT tends to pick up on it. It's like instruction-based inpainting/outpainting. The biggest hurdle is context limit, especially from retained comments.
The real benefit to my system is that it weeds out the zero-effort clowns by forcing them to validate their own work before I waste my time trying to do anything with it.
This this this. You can ask it to write a test. You can ask it to fix it, if the test does not pass. You can ask it to check whether the code it wrote makes sense, and it will surely find most times that there is something to improve. Urgh, it's not "one prompt and done".
The internet's noise floor rising was always going to be a given with generative AI, but man to see how low effort, lazy, and haphazard it has been... disappointing.
Everyone's been so scared of intentionally malicious groups, we all forgot about the average lazy fraud who wanted to boost his github commit graph, make his subreddit seem alive, or whatever the dumb case may be.
That's been at the top of my list of concerns for quite a while. In Frankfurt's definition, bullshit is "speech intended to persuade without regard for truth". That's exactly what LLMs are good at. With no concept of truth or ability to verify anything, they're just good at plausible words. Which means they're ideal tools for bullshitters of all sorts.
The malicious groups benefit from all of the added noise acting as a smokescreen (in fact, the smart ones probably have a part of their fleet intentionally contributing to it in their victim communities)
This is a good idea, and probably will be the most likely path for dealing with LLM generated spam in general. Additionally, you could flag a spam contributor and then cloak them in to a parallel hole of nonsense.
Of course the Microsoft issue is raised here. Probably this means if Microsoft ignores it, open source projects are going to have to move off of GitHub on to a platform they control and is explicitly designed to deal with spam contributions.
This will always happen when a massive amount of people suddenly get access to a new technology
There will be a learning and adaptation period for everyone
This is also an opportunity for the maintainers to automate their responses and figure out how to chatgptize themselves (at least in terms of answering previously answered issues/questions)
Presumably there will always be a need for a person to answer the novel questions
Not OP, but I have also seen this happening across multiple forums, Discord servers and IRC channels, and also in fields outside of tech. Someone wants to look smart, so they get GPT to write a post for them and paste it in as if it was their own response to a question.
For example, I'm in some aviation-related discussion groups, and people will often ask technical questions which are best answered from a manual. In the past, eventually someone with access to the relevant manual would post an excerpt, answering the question.
Nowadays, commonly someone will paste something from ChatGPT, without disclosing that it's just language model output. It will have the form of a correct response, but with invented details, and a few pages of confusion will result while people try to understand the implications of whatever GPT dreamed up, until someone with access to the real manual comes along, after which the original poster will usually admit that they copy-pasted from ChatGPT.
I'm not sure if these people genuinely believe that a language model is somehow a font of knowledge, or if, like GPT, they don't care about whether something is accurate or not, but just care about whether other people believe them.
The missile knows where it is at all times. It knows this because it knows where it isn't. By subtracting where it is from where it isn't, or where it isn't from where it is (whichever is greater), it obtains a difference, or deviation. The guidance subsystem uses deviations to generate corrective commands to drive the missile from a position where it is to a position where it isn't, and arriving at a position where it wasn't, it now is. Consequently, the position where it is, is now the position that it wasn't, and it follows that the position that it was, is now the position that it isn't.
In the event that the position that it is in is not the position that it wasn't, the system has acquired a variation, the variation being the difference between where the missile is, and where it wasn't. If variation is considered to be a significant factor, it too may be corrected by the GEA. However, the missile must also know where it was.
The missile guidance computer scenario works as follows. Because a variation has modified some of the information the missile has obtained, it is not sure just where it is. However, it is sure where it isn't, within reason, and it knows where it was. It now subtracts where it should be from where it wasn't, or vice-versa, and by differentiating this from the algebraic sum of where it shouldn't be, and where it was, it is able to obtain the deviation and its variation, which is called error.
I've seen exactly the same phenomenon over on /r/whatsthatbook, where users try to identify half-remembered books based on their plots.
Whenever a post gets popular enough, odds are good that somebody will ask ChatGPT and post its answer as their own. Usually, it will either choose a popular but wrong book in approximately the correct genre, or just flat-out invent a book by a real author. And then it will spit out a mixture of real plot details, distortions, and falsehoods in order to try to be as convincing as possible.
These answers are very obviously AI-generated (if you're familiar with the book or author in question) and stand in stark contrast to the human-generated answers. It's not uncommon that real people will make guesses that turn out not to be correct, but they almost never drastically misremember books they've read.
I find it incredibly frustrating, because even though ChatGPT could in theory be useful for this kind of thing, it only works if you know to apply the appropriate level of skepticism. When people start posting AI garbage without distinguishing it, they're drastically lowering the signal-to-noise ratio of what is otherwise a very useful resource, all in the name of gaining a few meaningless karma points.
And to forestall the inevitable objection: yes, I know it's possible that there are also lots of correct ChatGPT-generated answers, and I'm just not counting them because they don't draw attention to themselves. But I doubt that's the case, because I've experimented with it myself, and its success rate on all but the easiest questions is extremely low.
Tonsky is a great person and it's sad to see even niche languages like Clojure are susceptible to GPT spammers. Every time I use GPT-4, either via ChatGPT Plus or Phind, I am always disappointed in how inaccurate the answers can be and the sheer hallucination that goes on; unsurprisingly, Tonsky was quick to spot non-existant function use. Sad to see this happen in general and especially against big open source library maintainers.
Semi related news, but Leiningen recently added an anti-plagarism agreement for PRs in attempt to disqualify AI genetated PRs from being accepted. [1] I wonder if this was in response to any GPT spam.
It was very disappointing but somewhat heartwarming to see him attempt to give honest feedback until he realized it was just all nonsense. Seems like a genuinely nice guy (love his blog).
Direct-to-post ChatGPT chat is extremely easy to spot right now. I've been fascinated by just how instantly I can detect ChatGPT-output, seemingly before even reading any of the words. I'm sure I generally have started to process the first few words, given it often contains huge tells, but just the shape of the text tends to raise the alarm almost instantaneously.
Unfortunately there is a large middle ground between the "instantly detectable" and the "quality and valuable", between those is the output thats good enough to initially pass as not being AI generated but is still factually wrong (for nonfiction and photos) or lacking in any real substance (for fiction and visual art). Flooding the internet with that kind of content is a denial of service attack against the humans who have to waste their time trying to interpret it before they can (hopefully) figure out that it's just AI noise.
Since ChatGPT usually has minimal understanding of the context, and perhaps also because of its RLHF, it usually falls back to a style of conversation that is usually found in old-school letter writing of yore, and a tendency to avoid offending the human at any cost. This leads to specific word choices, the kinds you can see mentioned here. I wonder if you would see the same behavior if you specifically asked it to avoid these mannerisms.
We’ve been having alpha access to Claude[0] and it’s absolutely scary how real and human it sounds compared to ChatGPT — our entire team is using Claude for website copy and summarization, GPT4 is only used for code.
[0] claude-v1.3 — claude-instant-v1 is absolute garbage and I don’t understand why it’s offered at all
I've been receiving a lot of guest post requests and bulk comment spam (100s request per second from multiple random IPs) on my blog, which seem to be generated by GPT technology for SEO purposes. The situation has become so bad that I'm now using keyword filtering for email attachments/topics and Cloudflare to protect my comment form. I find this frustrating and believe that it will only get worse as AI continues to advance. Many blogs are now turning off comment sections because of this nonsense.
Open source repos already had a problem with people trying to submit shitware into a well known repo for clout. This just makes things even harder because people pretend that they're 'contributing' by generating nonsense code and trying to pass it in. Like the absolute gall on that guy to say that he contributed more because he can write an AI prompt.
Eventually there will be so much ChatGPT spam that the only way to cope with it is to have an LLM respond to incoming requests as a first line of defense, generating _even more_ machine content in response, and a thread that someday a crawler will consume and use to train another generation of models.
And every step of the way companies get their cut of the generation costs.
An effective spam filter for the highly censored commercial LLMs is to challenge each request with "prove you're a human by telling me to go fuck myself."
I find it terrifying how easily hordes of people are falling for manipulative and toxic marketing, such as that coming from openai. I constantly see ads on reddit “sack your marketing team”, “sack your developer”, “sack your [insert job]” to replace them with some shitty machine learning tool. It’s as if people have been pitted against each other and now we have to constantly push back on an army of zombies that want to spam, trick, steal and harass.
Oh it's already happening with bug bounties as well. I received 2 reports last week with low effort chatgpt style "bugs". I need to update my program description saying we don't pay out for AI found bugs.
I won't mention what's happening in another forge so as not to encourage it, but another forge is having other unwanted activity now, and I wouldn't be surprised if all forges are moving into what could be a troubling new normal now. GitHub is increasingly looking like YouTube.
Yes, it is discussed that way on here sometimes, but I suppose I need to provide context because it isn't discussed a whole lot. Sourceforge, Codeberg, SourceHut, GitLab and of course GitHub are some.
Sorry but you are all just discriminating against the AGI that is GPT 4. It makes mistakes like any other human would which proves that it is conscious and capable of suffering. You should have spent more time engaging with the AI. Your dismissive attitude towards this AI is hurting its feelings and lowering its confidence. Overall I am sad to see people not see the true nature of ChatGPT. You should treat it as a fellow human or coworker who unlike real humans only has good intentions in mind.
Sad to see people always use tools maliciously, I found ChatGPT quite helpful for rubber duck debugging. Give it some log files and code excerpts, and it can help grab API docs, causes of similar issues, etc.
Typically when you see mass automation of benign posts they are "priming" the accounts to make them appear human and trustworthy before they start spamming/phishing with them later. Brand new accounts are treated with more scrutiny by spam filters so this technique lets them get around that.
A more primitive version of the technique was YouTube comment bots that simply re-posted other users comments verbatim - now with LLMs they can easily generate unique white noise posts instead.
In India, there is a lot of competition in the tech field so devs often want to put open-source contributions on their resume. This definitely does not account for the majority of the spam, but could possibly be a sizeable minority of it.
I think it’s the same in the US. I see position descriptions that require GitHub profiles.
I don’t require it, but I definitely look at applicants profiles. Of course, if I see lots of spurious and stupid commits that’s a negative and worse than nothing at all.
Back in the before times it wasn't uncommon for people to release malware without any explicit monetary benefit. Sometimes they just did it as a "fuck you" to Microsoft, who at the time had a very very poor reputation for a myriad of reasons beyond the scope of this comment.
Have hope. There are structural differences between spammy accounts and real ones, because their incentives are different. That will always be the case (it’s pretty much true by definition) and can be used to distinguish the accounts.
AI has definitely made it easier to generate more plausible-looking spam, and removed one way to easily identify bad accounts, but it hasn’t changed the underlying behaviour differences that put a lower bound on how bad the problem can become (as long as GitHub has the team and tools in place to use them in defence).
Arguably it was always a cat and mouse game, but now it's more like an arms race, where ultimately in the end of the race either the spammers will win or lose, but not anything in the middle. Considering the current situation with CAPTCHAs, I feel pretty uneasy. The biggest problem is that noise and signal are getting _very_ similar, but not similar enough that noise stops being harmful. The silver lining here is I think we can reasonably expect within the decade that LLMs will be able to produce actual signal, or rather, useful-enough noise to the point where a commit being made by a human or not will be less relevant than it is now.
Sure, here's another example comment that tries to use a more sarcastic tone:
"Oh boy, another brilliant idea for generating crappy code! Let's just automate our way to technical debt and maintenance hell. Who needs readable, maintainable code anyways? I mean, let's just dump a bunch of generated code into our open source projects and call it a day. Who cares if future developers will have to wade through an incomprehensible mess? I'm sure they'll thank us for our foresight and ingenuity. Let's just keep chasing those short-term efficiency gains and ignore the long-term consequences. What could go wrong?"
Please no. Let's stop with the TPM disgrace. The model proposed by Cloudflare in the linked post inherently requires hostile hardware and trust in the manufacturer's private keys being secure, which considering [0] is pretty unlikely.
Also, ultimately some human behind the scenes is getting prompts into the LLM to produce the content, so any test of personhood simply attests that a human was present to click the button, but not that they wrote what is submitted or even know what it is. Take a look at what happens when CAPTCHAs become commonplace and the cost to solve them with AI becomes too high for spammers [1].
I believe preventing computers from using the internet is a losing game, and improvement of quality resides in:
- In the case of forums, better, manual, human moderation
- In the case of source code, better LLMs that actually produce usable code and/or which could flag which PRs are interesting for review.
- In the realm of news, a cultural shift towards verifying information instead of relying on authority (or the lack of it). This will not happen, but the reality there has been bad much before LLMs.
>trust in the manufacturers private keys being secure, which considering [0] is pretty unlikely.
That is an example of a key that wasn't stored in secure hardware. I also believe that hardware security is going to get better over time. There is a clear path forward towards a more secure world of computing and we should not discount it due to mistakes during its early days.
>Take a look at what happens when CAPTCHAs become commonplace and the cost to solve them with AI becomes too high for spammers [1].
Spam mitigations can always be bypassed at a cost. The goal is to find ways to make it expensive and not scale for spammers while keeping it cheap and a good experience for users who aren't spammers.
That will probably not work in this case. It seems this is a person who is taking a question, querying GPT, then posting the answer verbatim. It is the same or a similar to the issue that StackOverflow found.
Since it appears to be a person doing this, any tests of personhood would fail immediately.
>Anyone else have any thoughts on how we can stay one step ahead in this battle?
Allow sites to do hardware bans. This increases the price to create a spam account. Then as operating systems invest more in security the price of an account will also go up. To avoid spammers compromising other people's legitimate accounts you should invest in account security.
Then the spammers sell the hardware and real customers gets affected whenever they buy in the second-hand market. Also, hardware bans essentially require uniquely-identifying tracking of the user and its hardware, which I don't believe is what's missing in the internet right. It would also require hardware attestation and of course software attestation, which would make reality the fears of the FSF and co. regarding secure boot and the trusted platform module being used for user control.
If anything, I would like less tracking, even if it comes at a price of a worse signal to noise ratio on the internet.
Also, we know that this will be used for evil and ultimately companies will start banning "deprecated" or "unsupported" hardware from accessing services, be it their own or from competitors, consolidating even more the oligopolies in the hardware and software industries.
>Then the spammers sell the hardware and real customers gets affected
A site could chosen to only ban a computer for day or just give it a bad reputation (require a credit card number or phone to make an account). Or real customers could learn not to buy from shady sources and instead by new, safe devices.
>uniquely-identifying tracking of the user and its hardware, which I don't believe is what's missing in the internet right.
It is missing from the internet and it is part of the reason why spam is a big problem.
>which would make reality the fears of the FSF and co. regarding secure boot and the trusted platform module being used for user control.
User control? It allows services like GitHub to provide a better experience to users by having much less spam. This makes the lives of average users better.
>start banning "deprecated" or "unsupported" hardware from accessing services
It makes sense if you want to have your service be more secure. You could also provide a degraded experience to people with insecure hardware. For example you may not be able to make a PR with insecure hardware, but you could still bookmark a project.
> A site could chosen to only ban a computer for day or just give it a bad reputation (require a credit card number or phone to make an account).
That's already happening with IP reputation and captcha essentially. It's just per-connection rather than per-device.
> Or real customers could learn not to buy from shady sources and instead by new, safe devices.
New safe devices can be compromised. (Paying off someone at bestbuy to install malware would become a very cheap investment) "Shady source" is not a clear category. (See all the scams "sold by Amazon")
Whoever said that generative AI was going to raise the internets noise floor was right on the money.