
The AI That's Too Dangerous to Release - ReDeiPirati
https://blog.floydhub.com/gpt2/
======
6gvONxR4sf7o
One correction/clarification: It wasn't trained on reddit comments, but on
reddit links. So it's not learning to write like a redditor. It's learning to
write like what redditors read.

~~~
samstave
But ive been on reddit 13 years.

I extremely rarely upvote stories.

I read the comments section more than i read the articles.

I cant be the only one.

------
bArray
People write fake news all of the time, why is this AI so "dangerous"? At
worst it'll just add to an already existing mountain of news. At best I
believe it will prompt people to be more critical of what they read and where
they read it from, after all, how many times do you need to be embarrassed by
some auto-generated news sources?

~~~
bjourne
Spam. How on earth would a spam filter determine if such an auto-generated
text is spam or ham? Since it doesn't have a human brain it can't. So we get
into an arms race again. Right now, spam filters seem to have the upper hand
but that can change very quickly.

You can also use this text generation for less nefarious purposes such as for
submitting journal articles
([https://pdos.csail.mit.edu/archive/scigen/](https://pdos.csail.mit.edu/archive/scigen/)),
creating fake facebook campaigns, creating fake github profiles, phishing on
dating sites, fool Google's page ranking algorithms and so on.

~~~
bArray
To be completely honest, a lot of these services need a wake up call anyway.
Fake papers, fake Facebook campaigns, fake GitHub accounts, faking dating
profiles and crappy search rankings are already a problem - but these services
are currently unmotivated to solve these problems as they still work 99% of
the time.

We are already in an arms race for people's attention and data, that horse
already left the stable. Kindly asking people not to produce fake news is so
far not working out for us. This AI will be replicated, likely very soon, if
not already.

~~~
bjourne
I don't think your argument is very convincing since this problem would affect
almost every site on the internet and almost every text-based communication
channel. If burglary increased 100-fold, I don't think many would see it as a
wake-up call to install beefier home security systems.

~~~
bArray
> I don't think your argument is very convincing since this

> problem would affect almost every site on the internet and

> almost every text-based communication channel.

Every text based channel is already compromised, if not by AI, then by bad
actors. The only difference is that it's in lower volume than it might be with
AI. It's only a matter of time before somebody else recreates this AI (I'm
relatively sure I could with 2 solid months and some motivation) and the
longer we delay, potentially the greater the threat.

Consider this in terms of anti-virus software, do you respond to each
incremental advance in virus programming or do you wait until the problem is
overwhelmingly bad and you don't have years of incremental research to support
yourself?

> If burglary increased 100-fold, I don't think many would

> see it as a wake-up call to install beefier home security

> systems.

Even if this increased spam 100-fold, this wouldn't happen overnight. But by
locking it away, researchers can not actively work on a counter solution - so
when somebody invents a better AI and releases it, they are extremely ill
prepared. Not only this, the people themselves are unprepared too.

I think in delaying the handling this problem, the potential for mass
disruption increases - not decreases.

------
rossenberg79
For those who do not see the danger of advanced AI generated text being rolled
out everywhere, imagine a web where you have to _carefully read_ everything
you see in order to determine if it’s written by a human _before_ you can
start to take the text seriously.

That means no more skimming threads to get the gist of what people are saying,
no more skimming through answers on stackoverflow, no more skimming through
articles.

An article that looks reasonable on a cursory glance only begins to fall apart
when you spend (waste) time reading it carefully.

It also means anyone posting content must spend extra effort proving they are
a human for their readers.

~~~
samstave
Conversely, maybe the people who should be replaced by such AI are teachers
and professors.

What if you could beneficially weaponize an AI to teach children/people a vast
body of knowledge quickly on any given subject.

Please see the movie; Lawnmower Man

~~~
matt4077
Nobody is disputing that AI can be useful, so your comment seems to be missing
the point.

It would be slightly more coherent to at least point to a potential use case
for _text generation_ specifically.

------
lfnoise
Best sentence in the article: "It also happens to be trained on a large chunk
of Reddit, since the author decided that this was undeniably the perfect
location to obtain high quality, impeccable prose."

~~~
czr
While snappy, this sentence is misleading. The model is trained on _webpages
whose links have received at least 3 karma on reddit_ , not on reddit comments
themselves [0]. The motivation for this choice is not to find impeccable
prose, but rather to establish a lower bound on quality for the dataset, and
avoid training the model on too much of the spam/nonsense text that is
frequent in e.g. CommonCrawl. Also, the "author" is not a single person, but
rather 6+ people [1].

[0] [https://openai.com/blog/better-language-
models/](https://openai.com/blog/better-language-models/) [1]
[https://d4mucfpksywv.cloudfront.net/better-language-
models/l...](https://d4mucfpksywv.cloudfront.net/better-language-
models/language_models_are_unsupervised_multitask_learners.pdf)

------
ghobs91
Super clickbaity and low effort title.

~~~
jonathankoren
I agree. The hype is strong with OpenAI. The marketing of GPT-2 seemed
simultaneously braggadocio while also trying to capitalize on woke AI.

“We made this super cool thing. It’s super dangerous. You can’t see it.” I
mean, come on.

They can earn their reputation the regular way, and it will be fine. Sure they
have the click baity DOTA demos, and maybe they are doing some interesting
stuff. But OpenAI’s announcement of GPT-2 was particularly egregious.

------
Animats
Oh, it was trained on Reddit. That makes it clearer why it comes up with
coherent blithering.

~~~
parksy
If it was trained on the entire body of scientific knowledge (or you know, as
close as possible), would it be able to spit out useful correlations by
interpolating?

If trained on massive sets of source code and briefs (or perhaps a set of unit
tests), could it spit out functioning code based on a problem statement?

I'm just not well versed enough with ML to know if these things are a
possibility or if it will just remix its training data in probabilistically
correct, not-guaranteed-to-make sense ways?

~~~
Figs
Almost certainly not. It will generate things that _look_ like the text of
scientific papers on a casual glance, but which will be dream-logic gibberish
if you actually read it. The unicorn example makes no sense if you stop and
think about what it's actually saying. e.g. a "unicorn" that has _four_ horns?
Solves a 200 year old mystery... about something _just_ discovered? So close
you could touch their horns... while viewing them from the air? It doesn't
make sense if you actually think about it at all, but if you're not really
paying attention, the _style_ is like a news report.

It's _better_ at making up bullshit, but it's still bullshit.

~~~
derefr
I fully believe that this is 90% of what the human brain is doing to solve the
same problems, though. Brains just also have a constraint/filter layer,
determining which of these bits of "made up bullshit" should be allowed to
rise from the layer generating them, to the motor output layer.

Or, to put that another way, flipping the perspective around: our minds could
consist of an intelligent, analytical, but utterly _unimaginative_ agent, that
sits there listening to a stream of suggestions spewed out by a distinct
second agent, one that is "creative" but has no idea about the constraints of
things like physics. The brain's analytical agent filters this stream of
suggestions, taking notice of the suggestions that seem like they'll make the
world change in the ways it "wants"†; and then it does those.

† Or, according to modern perceptual-control research, the agent attempts to
predict the world that will occur a few seconds in the future, with a bias
toward predicting world-states the reward-system has annotated as being
rewarding; and then it looks at what motor commands it "would have" issued in
that hypothetical world, and actually issues those. The stream of suggestions,
in this model, serve as input to feed the generative model of potential world-
states; the executive agent then must notice whether the potential world-state
is a "possible" world or an "impossible" world (and whether the motor commands
required of it are "possible" or "impossible" inputs), and filter out the
"impossible" worlds.

Under this hypothesis, dreaming is the state when your executive responsible
for filtering out "impossible" worlds isn't online. So you just get a
continuous "impossible" world generated from the streamed suggestions of the
creative-but-stupid bullshit-generating agent, with nothing to tear it
down—just as seen in these generative AIs. As consciousness returns, the
mental predicted world-state is noticed to be impossible by the now-online
analytical agent, and _is_ torn down.

~~~
red75prime
> which of these bits of "made up bullshit" should be allowed to rise from the
> layer generating them

That will be a lot of bullshit to filter out, if such agent doesn't provide
more of less detailed description of what has to be generated. People with
Broca's aphasia probably demonstrate a part of such input.

~~~
derefr
Sure; the sequence of hypothetical world-states being constructed by the
prediction agent would likely be fed back as stimuli to excite features within
the generative network (which it would know associations for, since these
hypothetical world-states are stimuli of the same "type" as the real world-
state stimuli it was trained on.)

In other words, it could operate just as AI generative networks like GPT2 do,
first receiving training input/output pairs; and then later, receiving input
prompts and "completing" them by generating outputs.

------
johnwyles
From the example it looks like with next to no changes at all it would be able
to spit out a plethora of "fake news" articles, even if it's only source was
reddit.

~~~
yogthos
I find it bizarre that people worry about this since we clearly already
produce more than enough bullshit on our own without any need for AI.

~~~
holbue
AI scales much better, than human bs generators. I consider at least this
aspect concerning.

~~~
yogthos
There are 7 billion humans on the planet now, we don't have a problem
generating bullshit at scale. The same methods you'd use to discern AI
generated bullshit are already needed to evaluate current bullshit. If
anything I'd argue that this kind of tech becoming common place could get
people to start evaluating all news more critically.

------
andreyk
Pretty decent overview of the model, but did they really have to go with that
title...

IMHO the more interesting story with GPT 2 is the hype around it as well as
the (huge) backlash around its release. [self-plug coming] If interested,
check out this summary of that whole story:
[https://www.skynettoday.com/briefs/gpt2](https://www.skynettoday.com/briefs/gpt2)

------
vortico
Stupid question: Can I play with GPT-2 by entering my own prompts using a
website or downloadable project? Or is it not ready yet?

~~~
Voloskaya
You can, see:
[https://github.com/openai/gpt-2/blob/master/DEVELOPERS.md](https://github.com/openai/gpt-2/blob/master/DEVELOPERS.md).

But this is for a smaller version of GPT-2 (~400M parameters). Since the
bigger model (2B parameters) was deemed too dangerous by OpenAI.

~~~
vortico
Thanks! Attempting now.

------
sideaccount
Lex Fridman recently stated on the Rogan podcast that openai only open sources
a basic version of their ai tools, is this true?

~~~
criddell
Yes, it's mostly true. Everything will eventually be released, they are just
doing so in stages and are hoping to generate a broad discussion of the
technology and it's implications.

Some details are available here:

[https://openai.com/blog/better-language-
models/](https://openai.com/blog/better-language-models/)

~~~
gwern
There's two issues here. There's the models and there's the source code. The
two smallest models have been released now, but it's unclear if/when OA will
release any more publicly. They have released even less source code: all the
code people have been using to train & finetune GPT-2 models has been
implemented by third parties and OA has declined to release any code beyond
simple sampling code, with no hint of even considering releasing training code
in the future, much less releasing 'everything'. (And the sampling code isn't
even that great; top-k sampling, for example, instead of standard beam
search.)

------
yogthos
Obviously now that people have seen that this is possible it will be
replicated in short order whether they release it or not.

~~~
petters
Yes, but training it is very expensive.

~~~
yogthos
But that only needs to be done once. Also, it turns out that the current crop
of neural nets is incredibly inefficient
[https://www.technologyreview.com/s/613514/a-new-way-to-
build...](https://www.technologyreview.com/s/613514/a-new-way-to-build-tiny-
neural-networks-could-create-powerful-ai-on-your-phone/)

------
drivingmenuts
Is it too soon for a Butlerian Jihad?

------
jancsika
> It also happens to be trained on a large chunk of Reddit, since the author
> decided that this was undeniably the perfect location to obtain high
> quality, impeccable prose.

Hm, why in the world would they not train it on Sci-hub articles and generate
output for which we have plenty of domain experts who can judge the quality?

~~~
czr
I'm not sure if you're being tongue-in-cheek, but:

* Training on Sci-hub or book torrents would probably get them sued, so that's right out.

* The point of training the model on _reddit-linked-articles with 3+ karma_ (not reddit comments, as the article suggests) is to filter out worthless content (spam or non-text pages) while still getting a large, diverse sampling of human writing. They're training huge models and they need as much data as possible to do so.

* Every native english speaker is a "domain expert" for the purpose of evaluating the model's results. The point at which we need subject-matter experts to evaluate the quality of neural-net generated research papers is many decades away; the excitement around GPT-2 is that it can generate coherent English sentences _at all_.

