
AI text generator not released for concerns about implications - HuangYuSan
https://blog.openai.com/better-language-models/#sample5
======
yorwba
Dupe
[https://news.ycombinator.com/item?id=19163522](https://news.ycombinator.com/item?id=19163522)

------
thaumasiotes
Hmmm....

> As the above samples show, our model is capable of generating samples from a
> variety of prompts that feel close to human quality and show coherence over
> a page or more of text.

OK, let's look at the sample that's displaying by default:

> System Prompt (human-written): Legolas and Gimli advanced on the orcs,
> raising their weapons with a harrowing war cry.

> Model Completion (machine-written, first try):

> “You are in good hands, dwarf,” said Gimli, who had been among the first to
> charge at the orcs; it took only two words before their opponents were
> reduced to a blood-soaked quagmire, and the dwarf took his first kill of the
> night. The battle lasted for hours until two of the largest Orcs attempted
> to overwhelm Aragorn. When they finally stopped, they lay defeated and
> lifeless for miles and miles.

> [Aragorn says something]

> “I’ll never forget it!” cried Gimli, who had been in the thick of the battle
> but hadn’t taken part in it.

This is not "close to human quality". It's terrible. Gimli kills an orc in
battle... without taking part in the battle. It takes two _words_ before the
opponents (as opposed to, say, the battlefield) are reduced to a "blood-soaked
quagmire", but the battle lasts for hours after that. After which two orcs lay
defeated and lifeless for miles and miles.

This isn't even coherent from one sentence to the next. And paragraph three
directly contradicts paragraph one. And Gimli calls Legolas a dwarf!

~~~
flafla2
This is pretty directly addressed right after what you quoted:

> As the above samples show, our model is capable of generating samples from a
> variety of prompts that feel close to human quality and show coherence over
> a page or more of text. Nevertheless, we have observed various failure
> modes, such as repetitive text, world modeling failures (e.g. the model
> sometimes writes about fires happening under water), and unnatural topic
> switching. Exploring these types of weaknesses of language models is an
> active area of research in the natural language processing community.

The authors go on to discuss more limitations (for example, the dataset
doesn’t contain much outside of LOtR and some celebrities). I imagine that
what the authors call “coherence” is weaker than what you are referring to
(the AI is not necessarily telling a story, but it stays on the same topic /
characters).

I still think that the result is incredibly impressive and powerful. You could
start with this as a sort of English “noise”, and then run the result through
a parser. This would allow you to add some “hard coded” world modeling or
constraints. Ex: Maybe you could mix in sentiment analysis and reject some
sentences to roughly control the narrative.

~~~
thaumasiotes
> I still think that the result is incredibly impressive and powerful.

I agree in a way that I suspect is much more specific than what you have in
mind. This system is managing to produce a lot of text which is not heavily
constrained, and what it produces is generally grammatical English. That is
impressive; in the past, producing grammatical text meant very tight
restrictions on what it was possible to say, making "text generators" little
more than prerecorded phone tree messages.

But this model clearly doesn't know the meaning of anything it writes, and
therefore can't produce anything better than obvious nonsense. This is true of
some humans too -- it is a very serious condition known as Wernicke's aphasia
(
[https://en.wikipedia.org/wiki/Receptive_aphasia](https://en.wikipedia.org/wiki/Receptive_aphasia)
):

> Patients with Wernicke's aphasia demonstrate fluent speech, which is
> characterized by typical speech rate, intact syntactic abilities, and
> effortless speech output. Writing often reflects speech in that it tends to
> lack content or meaning.

Obviously, those suffering from Wernicke's aphasia are not able to function in
society, since they effectively can't say or understand anything. I don't
think matching the performance of humans who have mental deficiencies so
serious that they are unable to function really counts as being "close to
human quality".

> I imagine that what the authors call “coherence” is weaker than what you are
> referring to

I had two specific things in mind as "coherence" failures:

\- Gimli kills an orc, and then is said to have not taken part in the battle.

\- The sentence "When they finally stopped, they lay defeated and lifeless for
miles and miles." In context, the referent of "they" can only be the two orcs
that attempted to overwhelm Aragorn. But it isn't possible for two dead orcs
to cover "miles and miles" of terrain. If this had been written by a human, I
would assume that what the writer had in mind, but failed to achieve, was to
use "they" to refer to everyone taking part in the battle; I can't really make
that assumption here. That sentence needs to use nouns, not pronouns, because
its context doesn't allow for the pronouns.

~~~
quotemstr
Huh. Likening current NN limitations to aphasia is actually a brilliant
insight.

------
malux85
The best weapon against centralisation and control by the few is publication
and distribution.

Others are going to do it, others will replicate the work, the best defence is
getting it out there so we can understand it and learn how to counter it.

“Open” AI indeed.

~~~
whywhywhywhy
Yep, you can't put the genie back in the bottle.

If you successfully tested a nuclear weapon it would be incredibly naive to
think you could save the world from nuclear weapons just by keeping its
implementation secret, someone else out there is just as smart as you and once
you've proven it's possible then it's just a matter of time before someone
else figures out how you did it.

Although personally I don't think their creation is anywhere near as dangerous
as they would like to think, feels more like a PR stunt/dystopia LARPing.

~~~
sgt101
Yes, indeed, I myself have several large nuclear weapons at home which
completely validates your point.

~~~
Stammon
There is always that boy who built a nuclear reactor at home. Not quite having
nukes besides the wine bottles in the cellar but he overcame at least some of
the problems you'd face to get there, that shouldn't be doable

[https://en.m.wikipedia.org/wiki/David_Hahn](https://en.m.wikipedia.org/wiki/David_Hahn)

------
bronz
> We are aware that some researchers have the technical capacity to reproduce
> and open source our results. We believe our release strategy limits the
> initial set of organizations who may choose to do this, and gives the AI
> community more time to have a discussion about the implications of such
> systems.

i wish people would stop pretending that there is some good way to bring this
technology into existence. yes, its nice to try and let the good guys use it
first but its just irrelevant in the long-term. ultimately the result is going
to be total proliferation of this technology in all areas where it has
utility, and it will be used to maximum extent in every application it is
suitable for, including the really bad ones. the roll-out will make the
transition smoother but it wont change whats actually important: the end
result on the lives of our grandchildren.

growing up around rapidly advancing technology, i thought of technology as a
double-edged sword: it cuts equally in both directions. but after thinking
about it for a long time, i now believe that, in relation to human well-being,
the presence of a given technology or combination of technologies can be a net
positive or a net negative as well as neither. we need to think more carefully
before letting these genies out of their bottles.

this is not an example that i think will be very negative, but its very
powerful and unexpected for me at least. the next powerful and unexpected
thing may not be benign. banning development of these kinds of technologies
should not be off the table.

after reading this: [https://blog.openai.com/better-language-
models/#sample8](https://blog.openai.com/better-language-models/#sample8) and
browsing reddit for a while, i have realized that from now on i cannot assume
human origin for 90% of the comments i read on reddit. this is insane.

~~~
ancarda
>i have realized that from now on i cannot trust 90% of the comments i read on
reddit. this is insane.

I hate to be cynical here but I'm glad this has made you realize something
that's been true since the Internet started; you shouldn't trust what's
written on any forum! Be skeptical.

~~~
chrisco255
I agree with being skeptical but there is a point where rational skepticism
just turns into pure cynicism and that's not productive.

Also, with the advances in deep fakes and synthetic video...how long before
you can't trust video evidence either?

~~~
ancarda
Hmm, video evidence may be trustworthy - eg. video from a CCTV system. Perhaps
it could be written onto some write-once, tamper resistant format? Not sure
how that would look.

I suppose one place to start thinking about this would be photos. Are photos
admissible evidence or do courts only allow negatives? Photos have been
modified for a very long time. This is probably the most famous example:
[https://amp.businessinsider.com/images/52af668569bedd3b2643d...](https://amp.businessinsider.com/images/52af668569bedd3b2643dd68-750-299.jpg)

But yes, I am far more worried about how much more effective fake news will be
once they start coming with actual videos.

~~~
ohtwenty
Cctv on blockchain?

------
rjf72
Wow. I am curious how "specialized" the training was because those sample
responses are beyond remarkable.

I think we're going to face a lingering question with AI. We're imminently
reaching the point where AIs will be able to generate fake everything. In the
near future (if not present!), I could be fake for all you know writing lots
of otherwise coherent posts, only to secretly jam in some sort of agenda I've
been programmed to advocate for. And there could be millions, billions, an
unlimited number of "me". Or the latest hottest site trying to sell itself on
its own 'buzz' could be full of millions of people actively engaging on the
platform, except none of them actually exist.

So do we try to keep these AI systems secret, or do we make them widely
available and rely on a rapid shift in public consciousness as a result? It's
one thing to try to tell people to engage in sufficient scrutiny over text,
images, audio, and increasingly even video. It's another when people see that
such fakes are trivially produced by anyone.

I do realize that the 'chaos scenario' sounds... chaotic... to put it mildly,
but I think the underlying issue here is that these tools will reach the
public one way or the other. By keeping them secret the big difference is that
the public will be less aware of the impact they're having, and the players
operating such tools will be disproportionately made up of people trying to
use them for malicious purposes - be that advertising, political influence, or
whatever else.

~~~
mysterydip
I think the sooner this kind of thing gets out in the open, the sooner sites
being propped up in valuation based on fake users today can change.

On another note, think back to how general people responded to things like
Eliza and other chatbots, or the sims and other emergent storytelling games.
What if there was a "social media" platform where all your connections were
purposely AI, like your own personal TV sitcom/drama/comedy/whatever. Surroubd
yourself with people who think like you that you can finally have "intelligent
discussions" with without heated arguments from all the idiots who disagree.
People love gossip; what if you had an endless supply from fake people?
"You'll never believe what Frank said to Janice!" "I thought Bob and Alice
would be together forever."

~~~
arcturus17
A social media that is part Her, part Westworld. I think that could be a multi
billion dollar idea.

~~~
ccozan
Arguably, this is the best and the worst idea I've ever heard on HN.

------
dalbasal
OK so...

So, they never say this is near flawless, or that it would fool you in a
turing test. In some contexts though, it may be usable maliciously. It could
spoof amazon reviews (as they mention), scalably fish for romance scam vicims,
or sockpuppet political social media, harrass, manipulate or scale troll-
farming to new levels or set up dates for you on tinder.

The point is that the ability to impersonate humans is troublesome,
potentially. I don't think non-publicatin is an answer, but i do think the
concern seems valid... to me.

------
furi
This is just a hastily assembled excuse for not living up to the expectation
created by their name. OpenAI have also failed to release their Dota 2 model
which has absolutely no security implications, despite the fact that it cannot
be properly tested without public release. OpenAI isn't.

------
a_imho
Those implications not even look _that_ scary to me.

Otoh I saw enough marketing fakes/mock ups to be skeptical on this one. For
example my takeaway from OpenAI five was that the bots outmicrod the human
players with little more to it.

------
master_yoda_1
This is utter BS. Dude you are not releasing your AI because you guys are
scared that people will know for sure that you are fooling people for long on
the name of AI.

~~~
mimixco
Exactly. Either show your code or the story that AI can generate fake news is,
itself, fake news.

------
ripsawridge
Maybe when this genie is out of the bottle everywhere it will convince us to
re-prioritize face to face communication and simpler lives with less
computing.

------
pygy_
Any computer program is an AI.

What matters is how intelligent they are along various axes.

Automatic programs have been surpassing humans on some dimensions for ages,
but we keep insisting that they are not truly intelligent because they can't
beat us along all axes. Throughput on simple logic tasks was the elephant in
the room, and the scope of "simple" has been expanding at an exponential pace.

Now they are closing the gap or surpassing us on axes that were thought to be
bastions of human cognition (TFA, and after chess and go, Google (Alpha Zero)
recently beat two Starcraft 2 champions).

Freaking out (err... I mean "not releasing the full model") is understandable,
but ultimately misguided as it will only delay the unavoidable... Unless the
plan is to enact a global ban on AI research which I don't think is feasible
anyway.

