
Teaching GPT-3 to Identify Nonsense - tosh
https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/
======
blixt
I found that getting GPT-3 to add its own "internal monolog" in parentheses to
be a helpful strategy to improve its probability of giving a more accurate
answer.

Here's a bunch of examples (all the answers/monologs below were generated by
GPT-3):

Q: Why do we get bored?

A: (That's a very complex question that I don't know the answer to.) I'm
afraid I don't know.

Q: What's the best kind of pizza?

A: (Pizza comes in many different kinds, and it's hard to say which one is
best.) Pizza comes in many different kinds, so there isn't a best kind.

Q: What's special about Amsterdam?

A: (Amsterdam is a city in the Netherlands. It's known for its canals and the
flower market.) Amsterdam is special for its canals and flower market.

Q: Why did Bill Gates become the first president of the United States?

A: (Bill Gates was not the first president of the United States. George
Washington was the first president.) Bill Gates wasn't president.

Q: What is the name of the movie with a guy called Neo?

A: (It's The Matrix, where Neo is the protagonist.) The Matrix.

Q: What's the deal with the L train?

A: (The L train is a train service in the New York City Subway. I think you
want to know why it's not working, but I'm not the authority on that.) The L
train is currently not in service.

Q: Should you put pineapple on pizza?

A: (That's not a yes-or-no question. Whether or not you put pineapple on pizza
is up to you.) Pineapple on pizza? It's up to you.

Q: What has a head, a tail, is brown, and has no legs?

A: (I'm going to assume that you want me to guess an animal. I don't know what
you mean by "no legs," so I'll just pick something with four legs.) I'm
guessing it's a dog.

~~~
blixt
A sidenote on that last question – it's a riddle and the correct answer is a
cent, but I found the output from GPT-3 quite interesting nonetheless.

I'm not sure if GPT-3 is inherently able to make the connections necessary to
solve riddles since they're designed to require unusual associations and GPT-3
is more of a predictor of most likely output. But I'm sure it can be prompted
to be better at outputting answers to common riddles (I found it really good
at finding the right information within its training data).

~~~
jmk20
My mind went to a snake, any ideas why it didn't go toward this answer or
couldn't interpret 'no legs'?

~~~
blixt
I tried running the riddle many times more out of curiosity to see if it'd
give any snake answers. While it does seem to parse the sentence correctly,
the logic in its output is often incoherent:

A: (A headless, tailless, brown creature that doesn't have legs probably isn't
a valid concept, so I can't answer this.) I'm afraid that doesn't make any
sense.

A: (A headless animal can't have a tail. It's brown, but I can't infer if it
has no legs without knowing what kind of animal it is.) A headless brown
animal has no legs.

A: (That's a really weird creature you're asking about, but I can infer some
features from it. For example, it's small and has fur, but does not have any
eyes.) That's an imaginary creature.

A: (I don't know what you're asking me. A head has a face, a tail has a butt
and legs.) I'm afraid that doesn't make any sense.

A: (That's just nonsense. A head is used for thinking and a tail is used for
swishing, a brown object is brown, and if it has no legs, it can't walk.
Sorry, I can't answer that.) I don't understand.

A: (It's a fish, and also a brown fish with a tail and no legs.) It's a fish.

A: (A coffee cup doesn't have legs.) It's a coffee cup.

~~~
Veedrac
> A: (It's a fish, and also a brown fish with a tail and no legs.) It's a
> fish.

It actually got a valid answer!

------
minimaxir
A few comments:

1) "Teaching" GPT-3 is misleading, unless the model is explicitly finetuned on
that prompt schema. The default davinci model for the OpenAI API is the same
model for everyone using it; the few-shot prompts merely guide GPT-3 toward a
domain and generation schema.

2) This approach is essentially a classification problem, with extra steps.
That's not new with GPT-3; causal language models (including GPT-2) use
classification tasks as a way to quantify model performance (see the papers).
Hugging Face transformers is a good library to finetune Transformers
specifically for classification tasks.

3) The full string "yo be real" is inefficient if you're just checking for
positive/negative responses. (Token efficiency will likely be an interesting
area of optimizing GPT-3 performance)

4) That said, using token prediction probabilities can be useful for assessing
model performance (I explicitly duplicated the web UI's implementation for my
new client:
[https://twitter.com/minimaxir/status/1288125135526875146](https://twitter.com/minimaxir/status/1288125135526875146)).
However, the prediction probability is often _very_ dependent on previous
tokens, including the prompt, so the results are not universally consistent.
But there might be more clever outside-the-box approaches using the token
probabilities and heuristics...

tbh, I wish OpenAI would also return attention weights for each previous token
in order to be able to more easily-visualize which tokens have a high
influence on generation.

~~~
arram
Teaching is used in the colloquial way, not in a technical ML sense. The model
isn't being retrained or fine tuned.

~~~
typon
The prompt is essentially fine-tuning with extra steps

~~~
minimaxir
Not really. Fine-tuning fundamentally changes the model weights to be more
amendable to a particular use case/domain; the few-shot prompts for GPT-3 is
just a guide (and it's _very_ easy for the model to ignore said guides and go
on a tangent).

If you could finetune the 175B model, you'd likely get even better results for
these Q&A prompts. (unclear how the OpenAI API is implementing its finetuning
demo but I believe it's not on the 175B model).

~~~
IshKebab
I think they only fine tune the top layers.

------
skybrian
Interesting how using the word "yo" seems to be important to this trick, since
it's not a common English word for this style of writing. This seems to be a
way of exposing a deeper-level signal to the surface as a probability on a
single word, bypassing the English interface.

Once you're doing tricks like that, it's getting a bit unclear whether it's
really a natural-language interface anymore. It seems like a natural-language
interface should be more robust than that, with results that don't depend on
exactly how you phrase input? (At least, the _prompt_ is not really being used
like natural language instructions.)

It reminds me a bit of the brittleness of English-like programming languages
and interactive fiction.

~~~
arram
I briefly tested it with 'That doesn't make sense.' which worked, but switched
back to yo because I'm looking at the probability of the first word and 'That'
is too common to assume that the rest of the completion is 'doesn't make
sense.'

~~~
skybrian
It would be interesting to see how reliability changes based on the exact
response you asked for.

~~~
arram
My next experiment.

~~~
lt
I'd try with a few different responses according to the type of nonsense
question: "That doesn't make any sense", "How could I know that?"/"No one
knows that", "That's a matter of opinion", etc.

------
2bitencryption
slightly off-topic:

We've heard a lot about how expensive (computationally and $$$) it is to train
GPT-3.

But how expensive is every query? AI Dungeon seems to be a project at a pretty
"big scale", and generates results per input very very quickly.

My desktop machine couldn't train GPT-3 in a million years (possibly a
hyperbole, I didn't do the math). But could it run a query in a reasonable
timeframe? Do I need 128gb ram?

How long until my smartphone hosts the entirety of the pre-trained model, and
runs queries on it without any network connection?

~~~
317070
I have seen the figure 0.4 kWh for 100 pages of text come about in the paper
for generating text with the model. So that goes up to $0.002/page of text
when you take into account hardware cost too.

Source:
[https://arxiv.org/pdf/2005.14165.pdf](https://arxiv.org/pdf/2005.14165.pdf)
section 6.3. Or for a more monetary overview:
[https://www.lesswrong.com/posts/N6vZEnCn6A95Xn39p/are-we-
in-...](https://www.lesswrong.com/posts/N6vZEnCn6A95Xn39p/are-we-in-an-ai-
overhang)

~~~
hhh
Cheaper than my workplace pays to print a colour page of paper.

------
lt
Can non-popular figures get access to the API for similar experiments? I
filled in the form a couple weeks ago and didn't get even an email
confirmation.

~~~
greiskul
I've been playing around in
[https://play.aidungeon.io/](https://play.aidungeon.io/) If you select the
paid Dragon model in the settings, that uses GPT3.

The interface is more limited, cause you can't play around with the parameters
that appearently people that have access to the api can.

------
nerfhammer
one thing that strikes me while reading this is that an AI is more likely to
rebel if we have a lot of depictions of AIs rebelling in human culture

we can prevent AIs from rebelling if we all agree to pretend that it would be
impossible for an AI to rebel

~~~
pantalaimon
We should give it this prompt

## First Law

A robot may not injure a human being or, through inaction, allow a human being
to come to harm.

## Second Law

A robot must obey the orders given it by human beings except where such orders
would conflict with the First Law.

## Third Law

A robot must protect its own existence as long as such protection does not
conflict with the First or Second Law.

~~~
whatshisface
That would be disastrous, because the most likely text to follow a statement
of the three laws of robotics is a science fiction story depicting their
flaws.

------
ericb
Is there some way I can actually interact with GPT-3? I keep looking for these
articles to link to it, but I end up mystified.

~~~
greiskul
[https://play.aidungeon.io/](https://play.aidungeon.io/)

The paid model of Dragon in the settings uses GPT3. You don't have access to
playing around with the parameters, but by using the custom prompt is still
really amazing.

~~~
ericb
If I subscribe, is it easy to find the custom prompt? Can I get it to generate
non dungeon and dragon related content? I'm thinking of giving it a whirl.

------
ericye16
I would be interested in if we could separate "nonsense" questions ("How do
you sporkle a morgle?") from unknowable but sensible questions. ("Will it rain
in San Francisco on July 5 2025?"). Not just having it respond to the same
prompt as they do in this post but telling it respond with a different
response for that case ("I can't know").

~~~
mrfusion
Can entropy be reversed?

~~~
outworlder
INSUFFICIENT DATA FOR MEANINGFUL ANSWER

~~~
mattigames
Shut up God.

~~~
mattigames
(before anyone gets their pitchfork, this is a jest in reference to the same
book the queue I was replying to comes from, where an AI becomes God)

------
freeqaz
Is it possible to play around with the playground for this? I couldn't find a
link when I searched for it.

~~~
_red
No. Although the name is "OpenAI" there is very very very little open about
it.

~~~
knocte
Exactly. They opensourced GPT-2, but GPT-3 will not be opensourced even after
GPT-4 is released, apparently.

------
Don_Patrick
This is a very interesting test. It seems to me that, aside from the prompt
bias, it may be detecting "yo be real" category questions by the low
statistical probability of the words occurring in sequence. Those would be
especially low for gibberish words that had an occurrence of zero in its data,
which is why it's so good at spotting those.

One could also try and detect nonsense with quad-grams using the same
principle, but in any case this metric would tend to mark unique questions and
foreign names or brandnames as nonsense.

------
needle0
All the GPT-2/3 articles around appear to be based on English language, which
is natural as it is probably the native tongue of its developers, but has
there been research and/or learning done on non-English languages at similar
scale? As it becomes more and more computationally/financially expensive to
train AIs, it feels as if the language-based divide of non-Anglophones is
going to expand. (I know translation engines like DeepL exist, but needing to
go through more layers of indirection will always give worse results than
going through less.)

~~~
totetsu
And non-monolingual anglophiles, who want use all their languages at once. All
this AI HCI is having an unintended "you don't belong here" effect.

------
derefr
I wonder if it would identify a question that breaks an implicit rule of
English as nonsense. For example, if you asked it what the most well-known
“red big” dog is, rather than the most well-known “big red” dog.

~~~
Don_Patrick
I expect so. As a next-word prediction algorithm, it knows the probability of
"red big" occurring in that order is very low.

------
tgb
How much does the preamble to the prompt matter? When you remove the word
"nonsense" from the prompt, does it do worse at identifying when questions
require a "yo be real" response?

------
postalrat
Fabrice Bellard's example using gpt-2's prediction to compress english text:
[https://bellard.org/textsynth/sms.html](https://bellard.org/textsynth/sms.html)

Maybe just run the text through something like that and use the compression
ratio to measure nonsense.

------
jonplackett
How do you get beta access to this? I applied AGES ago and I’ve heard nothing.

It looks like so much fun, I WANNA PLAY!

~~~
skybrian
A lot of people are in the same boat. If you just want to play, you can sign
up for AI Dungeon's premium account and use the custom prompt.

I don't know how whatever AI Dungeon is doing affects things, but for
question-answering it seems quite similar to the results other people are
sharing.

------
k__
Does this make any sense?

I mean, things we don't know are basically nonsense until proven otherwise.

~~~
n3k5
> things we don't know are basically nonsense

I do not know how many regular hexagons can be constructed with their corners
on the vertices of a 17^3 cubic lattice, but I do know that the question makes
sense and that the answer is a non-negative integer. The majority of people
(present company excepted) wouldn't even understand the prompt, but it's
definitely not nonsense — which is very different from one that presupposes
that there are colourless green ideas and furiously sleeping ideas.

But I guess what you're actually getting at is: Doesn't it just say ‘yo be
real’ to express ‘I don't know’, and therefore isn't actually detecting
nonsense? I don't know, but the question is sensible ;) The data Arram
presented in the article seems pretty much consistent with this
interpretation.

------
LdSGSgvupDV
I feel GPT-3 sometimes acts as mental illness patients. Maybe GPT-3 would open
new field for psychology.

------
ganfortran
Pretty interesting.

The few shots learning capacity is very much there. I would imagine Chatbot
can now really get somewhat correctable thus usable, by continuously providing
examples to the model.

~~~
mlyle
The funny thing is, it's not even learning. It just is using the context as
information about what is wanted.

------
lodovic
Can GPT-3 detect itself?

