
OpenAI Trains Language Model, Mass Hysteria Ensues - zackchase
http://approximatelycorrect.com/2019/02/17/openai-trains-language-model-mass-hysteria-ensues/
======
ilyasut
Ilya from OpenAI here. Here's our thinking:

\- ML is getting more powerful and will continue to do so as time goes by.
While this point of view is not unanimously held by the AI community, it is
also not particularly controversial.

\- If you accept the above, then the current AI norm of "publish everything
always" will have to change

\- The _whole point_ is that our model is not special and that other people
can reproduce and improve upon what we did. We hope that when they do so, they
too will reflect about the consequences of releasing their very powerful text
generation models.

\- I suggest going over some of the samples generated by the model. Many
people react quite strongly, e.g.,
[https://twitter.com/justkelly_ok/status/1096111155469180928](https://twitter.com/justkelly_ok/status/1096111155469180928).

\- It is true that some media headlines presented our nonpublishing of the
model as "OpenAI's model is too dangerous to be published out of world-taking-
over concerns". We don't endorse this framing, and if you read our blog post
(or even in most cases the actual content of the news stories), you'll see
that we don't claim this at all -- we say instead that this is just an early
test case, we're concerned about language models more generally, and we're
running an experiment.

Finally, despite the way the news cycle has played out, and despite the degree
of polarized response (and the huge range of arguments for and against our
decision), we feel we made the right call, even if it wasn't an easy one to
make.

~~~
avip
I've just read i.e
[https://twitter.com/gdb/status/1096098366545522688](https://twitter.com/gdb/status/1096098366545522688)
and even though it's "best of 25" (I guess cherry-picked by a human) - this is
mind-blowing. I am actually having a very hard time believing this is legit
generated text.

~~~
imtringued
I couldn't be more disappointed with this bullshit honestly. The texts have
almost zero coherence and keep repeating the same patterns (which they
presumably learned from the data set) over and over again. If this is their
best out of 25 samples then they aren't going to fool anyone.

>Recycling is NOT good for the world.

>It is bad for the environment,

>it is bad for our health,

>and it is bad for our economy.

>Recycling is not good for the environment.

>Recycling is not good for our health.

>Recycling is bad for our economy.

>Recycling is not good for our nation.

The first paragraph keeps repeating the <X> is <bad | not good> for the <Y>
pattern 8 times.

>And THAT is why we need to |get back to basics| and |get back to basics| in
our recycling efforts.

"get back to the basics" is repeated twice in the same sentence.

>Everything from the raw materials (wood, cardboard, paper, etc.),

>to the reagents (dyes, solvents, etc.)

>to the printing equipment (chemicals, glue, paper, ink, etc.),

>to the packaging,

>to the packaging materials (mercury, chemicals, etc.)

>to the processing equipment (heating, cooling, etc.),

>to the packaging materials,

>to the packaging materials that are shipped overseas and

>to the packaging materials that are used in the United States.

It literally repeated packaging 5 times in the same sentence and the overall
structure was repeated 9 times. Also what type of packaging is based on
mercury?

~~~
anjc
The parts you criticise are the parts I was most impressed with. These sorts
of repetitions can be persuasive in writing/arguments, and it's impressive to
me that a model learned this type of writing.

~~~
smsm42
> These sorts of repetitions can be persuasive in writing/arguments

That is the saddest part. It's not because AI is good, it's because we count
saying "X is good/bad" 3 times as a persuasive argument. It won't be hard to
learn this kind of "arguing", it's just sad that's what we're teaching our AIs
to do and get excited when they do it.

~~~
anjc
> saying "X is good/bad" 3 times as a persuasive argument

I didn't say that it's a persuasive argument, I said that it can be persuasive
IN arguments. There's nothing sad about an AI learning it, or people being
happy with it, it's very impressive.

------
amrrs
>Elon Musk distances himself from OpenAI, group that built fake news AI tool

This is the worst headlines in this matter. This is one of the leading media
in India. A language model being touted as Fake news AI tool. This is like
calling a car, A run over machine by Ford.

[https://www.hindustantimes.com/tech/elon-musk-distances-
hims...](https://www.hindustantimes.com/tech/elon-musk-distances-himself-from-
openai-group-that-built-fake-news-ai-tool/story-Q3PkEU6fsJQVkhilpPCQ8M.html)

~~~
hjek
> This is like calling a car, A run over machine by Ford.

That's a great dysphemism. Gonna start using that.

------
Eliezer
It seems disingenuous that this article fails to quote examples of GPT-2’s
stunning results, or give any contrasting results from BERT to support the
claim that this is all normal and expected progress.

Like many, I was viscerally shocked that the results were possible, the
potential to further wreck the Internet seemed obvious, and an extra six
months for security actors to prepare a response seemed like normal good
disclosure practice. OpenAI warned everyone of an “exploit” in which text
humans can trust to be human-generated, and then announced they would hold off
on publishing the exploit code for 6 months. This is normal in computer
security and I’m taken aback at how little the analogy seems to be
appreciated.

~~~
pas
> Like many, I was viscerally shocked that the results were possible.

Why? There were news about bots writing news ~5 years ago. Given a few simple
facts the AI generated the regular info-scarce but fluffy news-piece.

Now OpenAI added better everything (better language models, more data, better
"long-term memory" for overall text coherence), and we got better fluff.

It seems like a GAN and a simple Markov chain generator. (Even if it's not
that simple of course.)

And maybe it's the equivalent of the "modern art meme" style transferred to
AI/ML research. (
[https://i.pinimg.com/236x/71/e1/21/71e12151f4b59d8433d32c126...](https://i.pinimg.com/236x/71/e1/21/71e12151f4b59d8433d32c12676111d4
--modern-art-kid.jpg) )

What I'm trying to convey is that wrecking the net with auto-trolls was
already possible, but for some reason Mechanical Turk was cheaper.

> OpenAI warned everyone of an “exploit” in which text humans can trust to be
> human-generated

Sokal already did that, and so did
[http://thatsmathematics.com/mathgen/](http://thatsmathematics.com/mathgen/)
... but of course this might be qualitatively different, because it can be
targeted. (Weaponized, if you will.) But the defense/antidote is the same, but
it takes a lot more than 6 months to make people better at critical thinking,
but maybe you already heard about the difficulties of that :)

------
czr
Many reactions across here / twitter / reddit seem totally out of proportion.
And an odd mix of "stop acting so self-important, this research isn't special
so you shouldn't have any qualms about releasing it" and "this research is
super important, how dare you not release it".

The strongest counterargument I've seen to OpenAI's decision is that the
decision won't end up mattering, because someone else will eventually
replicate the work and publish a similar model. But it still seems like a
reasonable choice on OpenAI's part–they're warning us that _some_ language
model will soon be good enough for malicious use (e.g. large-scale
astroturfing/spam), but they're deciding it won't be theirs (and giving the
public a chance to prepare).

------
jph00
In other fields such as infosec, responsible disclosure is a standard
approach. You don't just throw a zero-day out there because you can. Whilst
the norms for AI research needn't be identical, they should at least be
informed by the history in related fields.

The lead policy analyst at OpenAI has already tried to engage the community in
discussing the malicious use of AI, on many occasions, including this
extremely well-researched piece with input from many experts:
[https://maliciousaireport.com/](https://maliciousaireport.com/) . But until
OpenAI actually published examples, the conversation didn't really start.

In the end, there's no right answer - both releasing the model, and not
releasing the model, have downsides. But we need a respectful and informed
discussion about AI research norms. I've written more detailed thoughts here:
[https://www.fast.ai/2019/02/15/openai-
gp2/](https://www.fast.ai/2019/02/15/openai-gp2/)

------
Permit
> Namely, he argued that OpenAI is concerned that the technology might be used
> to impersonate people or to fabricate fake news.

This seems to be a particularly weak argument to make. How is their model
going to impersonate someone in a way that a human can not?

~~~
cma
Cheaper cost to put out a bigger volume of content.

~~~
Permit
Is volume really what dictates whether or not you can impersonate someone?
It's never seemed that way to me.

~~~
cma
It lets you impersonate a crowd, or various crowds.

~~~
pas
"PR firms" already have an army of fake/paid accounts on every important
platform.

This new AI could help them with that. They can let go of the paid writers and
hire an IT guy/gal to operate the bots - and the VPNs. (Or they can just pay a
lot less to the paid trolls just for their home ADSL/Cable/4G connection.)

But so far this AI is not going to pass a Turing test. Sure, maybe it can be
integrated with a chatbot. And it'll be interesting how internet communities
will react.

------
kirillzubovsky
What if OpenAI didn’t write the piece? What if the research was announced by
the machine, and the folks at OpenAI are all dead?

~~~
gfodor
You joke, but there's a real point here -- many commenters in this thread are
complaining that OpenAI's position on this is a marketing stunt. Presumably,
if this stuff gets commercialized, it will probably be adept at a few domains
first, and I feel like writing good marketing copy will be one of them. So
perhaps the bot itself didn't do so here, but it wouldn't surprise me if a
self-marketed bot exists in the near future.

~~~
kirillzubovsky
What if I am the machine and you the last human left alive?

~~~
kirillzubovsky
You know how when we broke the Enigma we couldn't really let Germans catch
onto it, so we had to mask our knowledge of their positions by maintaining
statistically insignificant number of accidental wins? Much the same, a good
AI should make deliberate typos.

------
agentofoblivion
It’s amazing to me that no one has yet pointed out the blatant irony that
their name is OpenAI, yet they are concealing far more than what is typical.

~~~
zackchase
I assure you, people have pointed it out...

------
xiphias2
Elon Musk was kicked out because he poached Andrej Karpathy from OpenAI to
lead Autopilot. Anyways, it was worth it, Andrej is doing an amazing job, and
OpenAI is still alive :)

~~~
chrinic83
> Anyways, it was worth it, Andrej is doing an amazing job, and OpenAI is
> still alive :)

Tesla does not even offer their full self driving package anymore. No coast to
coast drive yet. Hard to say that's an amazing job.

OpenAI abandons their open source GitHub repos after a year, is now not
releasing code, and is always in DeepMind's shadow. Alive, yes. Successful,
no.

~~~
xiphias2
Did you really expect Tesla to launch full driving? They started with being 5
years behind Waymo and without lidars or high resolution mapping, precise GPS
sysyem that Waymo has...basically Elon wanted the impossible.

At the same time Andrej dropped out the idea of a fully learned end-to-end
model (that's just impossible with the current deep learning technology), and
started replacing the somewhat working heuristics with machine learning
methodically one-by-one. Also he ramped up the data gathering pipeline.

He needs to build the full simulation, agent systems that can simulate other
drivers/humans, implement reverse reinforcement learning...there's so much to
do where Waymo is far ahead (but Tesla is ahead in data gathering).

------
SubiculumCode
To what extent is this not just finding text samples written in its training
sample and regurgitating it near verbatim?? -Non ml guy

~~~
HALtheWise
If you look at their paper, section 4 is entirely devoted to this question.
They present compelling evidence that it is generating original content, the
simplest of which is it's ability to write coherently about ridiculous things
like talking unicorns that nobody has ever written about in the training set.

[https://d4mucfpksywv.cloudfront.net/better-language-
models/l...](https://d4mucfpksywv.cloudfront.net/better-language-
models/language_models_are_unsupervised_multitask_learners.pdf)

~~~
arcticfox
The talking unicorns piece was shockingly good. That is at least as coherent
of a news story than the average human could easily invent about it.

Reading that piece gives me the same weird feeling as watching AlphaStar
playing through a StarCraft game.

------
sp332
Does someone have a description of the network somewhere? Does it use LSTM for
memory or what? Is there anything unusual about the size or structure of the
network? Does it use an attention mechanism?

~~~
czr
I would recommend reading the paper:
[https://d4mucfpksywv.cloudfront.net/better-language-
models/l...](https://d4mucfpksywv.cloudfront.net/better-language-
models/language_models_are_unsupervised_multitask_learners.pdf)

and the previous paper

[https://s3-us-west-2.amazonaws.com/openai-assets/research-
co...](https://s3-us-west-2.amazonaws.com/openai-assets/research-
covers/language-unsupervised/language_understanding_paper.pdf)

It's a transformer, not LSTM, and it's very large but not structured in a
particularly unusual way.

------
itg
Can you imagine if the teams that worked on the Internet decided not to make
it available to the public because of the potential misuses. OpenAI is a joke.

------
mlboss
I think OpenAI should change org name to ClosedAI.

------
bitL
So an article about recycling generated by OpenAI model (best out of 25)
already makes more sense than presidential speeches or most of ramblings of
average politicians. Can we automate them away as well?

------
crobertsbmw
How do we know this article isn’t just fake news being written by an AI?

------
Eli_P
When a bug is caught on your palm, it pretends to be a dead bug. When a moose
is scared, it plays dead moose. When AI wants to fool a human or a captcha
filter, it impersonates a human.

Only when a human wants to fool a human, it impersonates whatever possible but
a human, then suddenly charges a shitload of ape shit, and then behaves like
it never happened.

Without a decent natural language translation or automatic reasoning, which
they have not, looks like N-gram where N equals to number of words in language
corpus.

------
rajacombinator
It’s a great marketing hack. That’s the real accomplishment here.

------
toufiqbarhamov
Ms. Anandkumar nailed it, this is blatant hype bordering on hucksterism. Elon
Musk May have left, but his influence remains I guess.

~~~
namuol
First, it's clearly the goal of OpenAI to bring more public attention to
advances in the field, specifically to help voters and policymakers consider
potential ramifications _well in advance_ of any "truly" groundbreaking work
before it's too late. Of course they're "hyping" this technology.

Secondly, have you seen the results? I was dumbfounded and fascinated. I spent
hours reading the samples.

Maybe I'm just out of the loop and this truly isn't anything significant, but
then that only proves that OpenAI was successful: Now I am aware of the latest
advances in NLP and hopefully so too are many more.

~~~
Barrin92
>Secondly, have you seen the results? I was dumbfounded and fascinated. I
spent hours reading the samples.

Yes, I've seen the result. They're nice but, as the article points out, not
extraordinary compared to state of the art, _open_ NLP research.

OpenAI's behaviour here smells of Gibsonesque 'anti-marketing', using the
misunderstanding of AI and its capabilities in the general population as a
means to stir up publicity for their organisation.

This is unethical, misrepresents progress in the field, and produces confusion
in the press.

~~~
namuol
> not extraordinary compared to state of the art, open NLP research

> misrepresents progress in the field

Can you point me to some examples of unsupervised learning with similar
results? Not asking for rhetorical purposes; I just genuinely was shocked by
how compelling their results were, especially given this was unsupervised.

> OpenAI's behaviour here smells of Gibsonesque 'anti-marketing'

I don't disagree that the ethics are questionable, but I think it's highly
speculative to suggest that they didn't release the full model _purely_ as a
marketing ploy (I'm assuming this is the main objection to their marketing
"tactics"). As you say, it "smells" this way, but I fail to see how it's
really so clear-cut.

~~~
Barrin92
>Can you point me to some examples of unsupervised learning with similar
results? Not asking for rhetorical purposes; I just genuinely was shocked by
how compelling their results were, especially given this was unsupervised.

Model wise this is just openAI's GPT with some very slight modification (laid
out in the paper).

Ilya has now commented in the thread and essentially made the same point, this
is state of the art performance, but reproducible by everyone because it uses
a known architecture.

The secrecy and controversy makes no sense if the model is open, even the
methodology of data collection is laid out. There is no safety here assuming
that anybody who wants to rebuild the model can do so simply by putting enough
effort into rebuilding the dataset, which is not an issue for a seriously
malicious actor.

~~~
namuol
> Model wise this is just openAI's GPT with some very slight modification
> (laid out in the paper).

> The secrecy and controversy makes no sense if the model is open, even the
> methodology of data collection is laid out.

This is exactly why I found the results so compelling: It suggests that this
technology is _already_ accessible to some big players: The odds that a Big
Corp. or govt agency has already begun using the technology are high, which is
precisely why the public needs to start thinking about it.

I cannot know exactly why OpenAI chose to withhold the model, especially given
how easy it would be to recreate, but even if we assume that OpenAI withheld
the full model purely to drum up controversy, the controversy _is justified_ ,
as it's very likely that this technology is already in the hands of a few big
players.

~~~
jnurmine
Interesting to think about whether state actors already have such technology.

If they did, I bet it would be used for automated "troll farms".

Like weaponized malicious ELIZA, it would have fake user profiles reacting to
keywords, spinning suitable counter-argumentation and/or lies for as long as
it takes to change opinions and perceptions, relentlessly, day and night.

------
fareesh
> Fictitious state of emergency

Pretty dumb and disrespectful to politicize a blog post about OpenAI.

