I've started working on a version of GPT-2 which generates English text. The purpose of this is to improve its ability to predict the next character in a text, by having it learn 'grammatical rules' for English. It already works well for predicting the next character when it has seen only a small amount of text, but becomes less accurate as the amount of training text increases.
I have managed to improve this by having it generate text. That is, it creates an 'original' piece of text about 'topic x', then a slightly altered version of this text where one sentence has a single word changed, and this process is repeated many times (about a million). It seems to quickly learn how to vary sentences in a way that seems natural and realistic. I think the reason this works is because it reduces the chance that the grammar it has learned for one specific topic (e.g. snow) will accidentally be transferred to another topic (e.g. dogs).
Of course, this all means nothing unless it actually learns something from the process of generating text. I haven't tried this yet, but the plan is to have it generate text about a topic, then have a second GPT-2 system try to guess what that topic is. If the resulting system is noticeably better at this task, then we know the process has increased its ability to generalize.
One potential issue with this approach is that the text it generates is 'nonsensical', in that it is almost like a word-salad. Although this is a standard problem with neural nets (and other machine learning algorithms), in this case the text actually is a word-salad. It seems that it has learned the rules of grammar, but not the meaning of words. It is able to string words together in a way that sounds right, but the words don't actually mean anything.
Plot twist: This comment was generated by GPT-3 prompted with some of the comments in this thread.
The thing that kills me is that to the vast majority of human beings the nonsensical technobabble above is probably indistinguishable from real, honest, logically consistent technobabble.[a]
Soon enough, someone will replicate the Sokal hoax[b] with GPT-3 or another state-of-the-art language-generation model. It's not hard to imagine GPT-3 writing a fake paper that gets published in certain academic journals in the social sciences.
It's not hard to imagine GPT-3 writing a fake paper that gets published in certain academic journals in the social sciences. And then, it'll be all over for us. We won't have any more funding and our jobs will disappear. I can already hear the protests: "But we're not just scientists! We're also philosophers!" Well, yes and no. Philosophers are supposed to think about things philosophically, but they don't actually do anything about them; they're just entertainers. Scientists do something about them. They make things happen. And when those things happen, people take notice. If science and technology have a weakness, it's that they work too well. This was probably a strength at one point, but not anymore. In the not-too-distant future, there probably won't be any more philosophy professors; there will just be philosophers. But only in the same sense that there are lions and mushrooms.
Really? I got about halfway through and realized that the comment had no point. If you tried to summarize what it was arguing, beyond the first sentence, I don't think you could make a coherent summary.
Maybe the real lesson is we don't expect human-written comments on discussion fora to be particularly coherent....
Also both comments made me suspicious half way through and I scrolled to the bottom to check for a GPT-3 note. Without that note I would definitely have regarded it as incoherent rambling by a human.
Especially the second comment can be coherently interpreted with some good will and a cynical view of the humanities and philosophy. The "author" could say that once GPT-3 can write humanities papers it will quickly make humanity scientists redundant and that humanities scientists are philosophers is not important and doesn't warrant a job alone ("they don't actually do anything"). Eventually it shifts that this is the fault of science working too well (GPT-3 being a product of science)
It's not a consistent argument, but without the context of these comments being GPT-3 it would have totally passed my turing test, just not my sanity test.
I think (slash worry) that this is going to be a simple upgrade in future iterations. Obviously there are powerful attention mechanisms at work to keep the subject matter coherent, but we’re not too far off the model being able to generate a core thesis, and ensure that all the generated text supports that in some way.
I think that if that worked it would prove that either language is a much more powerful tool than we realize, or our cognitive capacities are much more trivial than we realize.
The model fundamentally has no understanding of the world, so if it can successfully argue about a central thesis without simply selecting pre-existing fragments, then it would suggest that the statistical relations between words capture directly our reasoning about the world.
Who here thinks some Donald Trump's answers were written by an early version of GPT3, designed to produce more bombastic and rambling rhetoric than usual?
In principle it’s not too far fetched ... there’s almost certainly some kind of data-driven algorithmic processing going into a lot of speech writing these days; some of the drops are so implausible they’d almost certainly have to have been suggested by a machine!
Not being sarcastic, but I know some people with less coherent writing than this. A lot of people struggle to make a point, use vague language, or wander in and out of topics quite easily.
It felt like it was making a slightly ranty observation that scientists are already trying to much to be philosophers than to actually do science that changes the world, yet science has brought us far enough that it acts as an enabler for all kinds of pop-philosphers.
The final bit doesn't quite connect, but overall I've seen far less coherent comments written by humans on subject with far more logical flaws.
I would not have imagined it was automatically written. Rambling and there's little connection between the first part and the latter, but absolutely something that might appear on a random internet forum.
Given that I know this stuff is generated text, it looks pretty good. But, if I’m judging it assuming that it was written by a human, it has a very uncanny valley sort of feel. That’s actually a good thing compared to previous models that would generate a lot of jarring non sequitors, because the GPT-3 text is very good if you look at it in 2-3 sentence chunks.
You say it like the bot wouldn't fit right in alongside most human comments because it meanders and doesn't seem to actually be responding to anyone, rather listening to itself talk.
Maybe the real lesson is it was trained on human-written comments in discussion fora, so it perfectly mimics the average lack of point, weak arguments, rambling and incoherence in fora?
It would be interesting to see if the output has a similar quality when trained only on highly regarded texts.
I don't think these gpt3 comments will get many upvotes on HN anyway. I downvoted the first one for being incoherent, but then realized it was meant as an example so I undowned it.
In possibly an unwise move, I'm actually going to respond to the initial point here.
There's a totally valid discipline in taking concepts from different areas and smushing them together to make a new idea. That's what a lot of creativity is, fundamentally. So a bot that's been trained across a wide variety of texts, spitting out an amalgam in response to a prompt that causes a connection to be made, is not only possible, but likely a very good way of generating papers (or at least abstracts) for humans to check. And if the raw output is readable, why not publish it?
That's a good question, how do we get access? I signed up on the list, but it must be thousands of people long by now. Does anyone here know anyone who can get people access?
I'm starting to sense that, in most scenarios, I will no longer want to engage in text-based conversations unless I know that I'm talking with a human. I already don't like spending a lot of time arguing with a bot on Twitter, this just makes it much more likely I'll also argue with a bot on medium-length text areas (e.g., HN, FB, Whatsapp, SMS, etc.) and maybe even on long-length text areas (e.g., Medium, blogs, NYTimes or things pretending to be real newspapers, etc.)
Second, I'm curious/terrified at how future iterations of GPT-3 may impact our ability to express ourselves and form bonds with other humans. First it's text messages and comments. Then it's essays. Then it's speeches. Then it's love letters. Then it's pitches. Then it's books. Then it's movie scripts. Then it's...
TLDR; Fascinated by the technology behind making something like this work and quite worried about the implications of the technology.
There was an article recently about people pursuing romantic relationships with chatbots. I thought there was a big HN discussion about it, but the only thing I've been able to find is this WSJ piece
The thing is that this thing has now crossed into the uncanny valley. Earlier it would have great trouble making a single sentence that makes sense. You only ever remember whether the last two sentences made sense and go together. And with GPT-3 any pair of sentences always makes almost perfect sense. By the time you're four sentences down you go wait a minute ...
This was very apparent when reading the generated stories [1].
Especially the shoggoth cat dialogue, I found that one really creepy. The fragment below comes straight out from the uncanny valley:
Human: Those memes sound funny. But you didn’t include any puns. So tell me, what is your favorite cat pun?
AI: Well, the best pun for me was the one he searched for the third time: “You didn’t eat all my fish, did you?” You see, the word “fish” can be replaced with the word “cats” to make the sentence read “Did you eat all my cats?”
Yeah, GPT-3 never really gives any kind of answer to “why”. It rambles on like Abraham Simpson talking about how an onion on his belt was the style at the time. Devoid of purpose it fills the void with meaningless words until told to stop. It’s subtle gibberish... and fucking irritating as soon as you catch it.
If he didn't put in the last line ("plot twist ...") I'm pretty sure no one here on HN would have guessed it.
In fact, while reading that comment I started to wonder why no one has tried to use GPT to generate text one character at a time. Or if someone has, what are the advantages and disadvantages over the BPE approach.
I’ve not studied neural nets and ML since undergrad level in 1998. So I am almost as knowledgeable as a random person on the street.
The quality of writing was very high, so I was convinced I was reading something put together by a human with agency... except it didn’t pass my gut-feeling “how IT works”. It made me suspect that either the algorithm (the described one, not the AI responsible) was off, or that I just didn’t understand AI any more. As I know I don’t have up to date AI knowledge, the algorithm appeared more believable. I hiked deep down the uncanny valley with that one.
I assume both previous comments, and this one, are also GPT-3?
Edit: it is amusing to think that soon the way to distinguish them will be that human comments have weird errors caused by smartphone keyboard "spell checking" in them...
On a re-read, I'm not sure why I thought that, sorry. Context: I don't know much about machine learning and when I was scanning through comments, doing text generation one character at a time seemed silly and I must have been in the grips of "everything could be GPT!" hysteria. My robot detector needs work, clearly. Need to get educated.
Well, you'd need to train it from scratch to operate on character level. And it would have smaller context, thus lower quality. So if you want same quality, you need much bigger context.
Still, would be an interesting experiment. Gwern swears it would improve stuff, so worth trying and comparing, I guess
There's actually precedent for this - sci-gen got papers accepted in IEEE and Springer publications, through peer review, and they had to investigate their back catalogue to look for others.
Given the propensity for academic writing to often favour the strategy of confusing the author through obfuscation (to make a minor advance sound more significant than it is), I suspect tools like this could, as you say, actually get published papers in some fields like social sciences. In an engineering or science paper you can check equations match conclusions, and that graphs match data etc.
In a more qualitative field of work, reviewed in a publish-or-perish system that doesn't incentivise time spent on detailed reviewing, I think there's a very real risk babble like this just comes across like every other paper they "review".
I think it takes a certain level of confidence to dismiss others' work as nonsensical waffle, but sadly this is a confidence many lack, and they assume there must be some sense located therein. Marketing text is a great place to train yourself to recognise much of what is written is meaningless hocum.
Dr. Herbert Schlangemann (fake alias for Sci-gen) not only got papers accepted in journals, it was invited to participate as session chair at conference co-sponsored by IEEE.
In a similar way to how image detection networks appear to key largely on visual textures, GPT-3 seems to key on jargon, tone and pacing, the texture of speech within specific milieus.
The thing that kills me is that soon enough, a "fake" paper written by GPT-3 will get published in an academic journal because it has actually contributed a new insight.
It's easy to consider text generation models as "just mimicking grammar". But isn't grammar also just a model of human cognition?
Is GPT modeling grammar or is it modeling human cognition? Since GPT can ingest radically more text (aka ideas) won't it soon be able to generate texts (aka ideas) that are a more accurate collation of current knowledge than any individual human could generate?
My impression is that these models are already doing far more than what the language production machinery in our brain does. We are able to produce language according to grammar and semantics, but we also have independent mental representations to guide the generation of language and to provide context.
I don't really understand why we're trying so hard to build models that can generate coherent texts based on having predigested only other texts, without any other experience of reality. Their capabilities appear already superhuman in their ability to imitate styles and patterns of any kind (including code generation, images, etc.). It feels like we're overshooting our target by trying to solve an unsolvable problem, that of deriving the semantics of reality from pure text, without any other type of input.
One of my favorite conspiracists, Miles Mathis, has this quality. He can strong together entire pages of very real and consistent nonsense that is totally logical and makes enough sense to be real. I have to remember I'm not reading a legit theory and now really do confuse myself with his version of science vs reality.
Kind of annoying 'hoax', though. Obviously you can publish garbage in fringe journals if you leverage pre-existing prestige and position like Sokal did. Doesn't really say anything about the social sciences.
You can also publish a lot of nonsense in certain chinese journals that optimize for quantity in quality, in whatever field you want.
Worse, Sokal's Revenge is probably inevitable, in which someone will generate a nightmarishly complex mathematical proof that takes the whole field in unexplored directions. Some of the most respected professors and most promising students will then be distracted for years trying, and ultimately failing, to make sense of it.
Some say this has already happened. Nobody has ever seen the Social Text editors and Mochizuki in the same room together, have they?
> Plot twist: This comment was generated by GPT-3 prompted with some of the comments in this thread.
This kills the forum.
Seriously, once this is weaponised, discussion of politics on the internet with strangers becomes completely pointless instead of just mostly pointless. You could potentially convince a human; you can't convince a neural net that isn't in learning mode.
It's more insidious than that. You can think you've convinced a human whereas you've just spent your energy on a bot. Assuming "political arguments on social media" has any relevance to voted cast, that's a vote for your side which doesn't happen.
That’s interesting, as within a sentence I had dismissed your comment as rambling and moved on to the next one, without thinking it had been generated... but maybe you’re double bluffing me.
Same, except that I skipped straight to the last line to check whether it was a generated text after I noticed the first sentence made no sense (GPT-2 already generates grammatically correct English sentences).
One potential issue with this approach is that the text it generates is 'nonsensical', in that it is almost like a word-salad. Although this is a standard problem with neural nets (and other machine learning algorithms), in this case the text actually is a word-salad. It seems that it has learned the rules of grammar, but not the meaning of words. It is able to string words together in a way that sounds right, but the words don't actually mean anything.
Plot twist: This comment was generated by GPT-3 prompted with some of the comments in this thread.