Hacker News new | past | comments | ask | show | jobs | submit login
AI-Generated Garbage Is Beginning to Pollute Scientific Literature (unite.ai)
45 points by Ice_cream_suit on July 20, 2021 | hide | past | favorite | 13 comments

"The researchers hypothesize that GPT-style generative models, and similar types of language generation frameworks, have been used to produce much of the text in the flagged papers; however, the way that a generative model abstracts its sources makes this difficult to prove, and the chief evidence lies in a common-sense evaluation of poor and unnecessary synonyms, and a meticulous examination of the submission’s logical coherence."

This sounds extremely unlikely.

First, scientific fraudsters are typically extremely lazy; they do not use fancy methods like writing generative models of fake data to ensure the numbers, say, add up to 100%, much less neural nets. (Do we believe that someone would go to the effort of setting up GPT to generate fake papers and then be satisfied with results like 'profound neural organization'? Easier to write such garbage by hand!)

Second, the given examples are so bad I'm insulted on behalf of GPT: 'profound neural organization' looks like a word-by-word calque using a thesaurus or particularly antiquated machine translation (in a "the vodka is good, but the meat is rotten" way), no contemporary AI involved; GPT would not generate such garbage.

Third, if GPT was generating these papers, it would be easy to detect by looking for top-k signatures, using GPT-targeted classifiers (they get >90% accuracy), or looking for stylistic tics of GPT; that they don't provide any specific GPT-like aspects of the text suggests there are none.

"Do we believe that someone would go to the effort of setting up GPT to generate fake papers..."

I'd say it depends on what their desired outcome is.

If they know that other people are using AI to assess the literature about a particular topic, then the content only needs to be good enough to pass that analytical AI's sniff test and bias the corpus in the way that the owner wants it to.

If the owner is lazy, perhaps that affords some time against this kind of pollution. But the costs and level of effort involved will continue to reduce.

The examples of "failed attempts at creative synonyms for known phrases in the machine learning sector" seem the output of translating the text multiple times (Language1->Language2->Language1), more than using a language model.

Why would GPT-3 or another language model use "profound neural organization", which is a non-existing term that it has never seen during training?

I agree. It looks like bad translations.

Here in Argentina we use a lot of loaned words from English for technical terms, sometimes I don't know which is the official translation to Spanish. But in Spain, they like to translate everything, so sometimes reading technical stuff in es-es is difficult. And if you blindly translate from es-es to English, you may get surprising results.

I sometimes write an early draft of a paragraph in es-ar, and then autotranslate it to English. The autotranlation handles the easy part, like word for word substitution and reorder noun-adjective to adjective-noun, but it needs a lot of further fixes to get a good translation. Technical words are sometimes messed, so it's important to double check them. Also, errors that are small typos in Spanish sometimes are translated to very things, that also cause weird results. (For example "cOsa"->"thing", but "cAsa"->"house".)

It is likely transation, not GPT, but

1. translation was considered a major axchievment in "AI" 10 years ago (not sure if it has "models", but I suspect yes)

2. are they sure its spinning (copying content via paraphrase/synonym) and not just bad translation from original Chinese?

This is a really interesting article, but I can’t ignore my pet peeve:

It has a section titled “Not Just Semantics,” but of course they really mean “syntax.” Semantics is meaning!

Clearly an Easter egg: an example of an incorrectly- and automatically-chosen synonym.

I absolutely love the way GPT-3 re-words common phrases. Examples from the article:

artificial intelligence (AI): ‘(counterfeit | human-made) consciousness’

ant colony: ‘subterranean insect (state | province | area | region | settlement)’

It's so clinical and literal. It reminds me a lot of the way Overwatch phrases things in Half-Life 2. For example, when you use a swarm of antlions (giant alien insects) to stage a prison break, Overwatch says

"Overwatch acknowledges critical exogen breach. Airwatch augmentation force dispatched and inbound. Hold for reinforcement"

"airwatch augmentation" sounds much more ominous than "air support".

I would be interested to see what GPT-3 could come up with if explicity tasked to do this type of re-phrasing. What happens if you ask it to pretend to be an alien intelligence?

> I would be interested to see what GPT-3 could come up with if explicity tasked to do this type of re-phrasing.

First, this is definitely not GPT-3, see my other comment. However, as one would expect, GPT-3 is capable of all sorts of fascinating 'rewriting' tasks. GPT-2 could do summarization via 'tldr' but GPT-3 can so so much more!

I enjoyed "overcomplicated explanations" ( https://www.gwern.net/GPT-3-nonfiction#overcomplicated-expla... ) as the reverse of 'explain it to a second grader' ( https://andrewmayneblog.wordpress.com/2020/06/13/openai-api-... ) but it can also do memes ( https://www.gwern.net/GPT-3#navy-seal-copypasta ) and literary style parody rewrites ( https://www.gwern.net/GPT-3#literary-parodies ), as well as 'emojification' ( https://aidungeon.medium.com/introducing-ai-dungeon-translat... https://andrewmayneblog.wordpress.com/2020/06/24/open-ai-alc... https://arxiv.org/abs/2106.13884#deepmind ) or screenplay ↔ story ( https://andrewmayneblog.wordpress.com/2020/07/08/openai-api-... ).

I haven't tried but I would expect the literary style parody prompts to work just as well if you dropped in "alien robot" or something like that, giving you text style transfer.

This is as pointed out translation spam.

Which uses whatever AI the big translation services use.

What's surprising is this is many years old on the web but I'm not sure it's been written up?

Massive 'news' blogs use it. Most stories will get a translation spam version.

Basically it allows copying without obvious copyright issues and might get great SEO. Not sure how on top of it Google is, some articles are saying Google can't detect it.

Even if Google is on top, Bing mightn't be able to find Google translator spam without Googles help/breaking Google's TOS

It could be worse. What if our AIs generated high quality papers?

I mean, that is the goal, isn't it?

Just imagine the increase in productivity per capita!

I look forward to viewing the mental gymnastics routines that will be required to keep up the facade that we still live in a scarcity-based economy.

we could just move that information onto a site where scientists have to verify their identity, we could also build a science blockchain and verify identities that way as well

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact