
Generating Fake Conversations by Fine-Tuning OpenAI's GPT on Data from Messenger - Tenoke
http://tenoke.github.io/blog/gpt-finetune
======
personjerry
Sorry, but these generated conversations seem nonsensical, nothing like the
OpenAI results.

~~~
Tenoke
It uses their small model and a tiny dataset in comparison (and a small amount
of training). It is more showcasing how much it learns (and doesn't learn)
with those limitations in place. As well as allowing you to recreate it with
perhaps a few minutes of work and less than an hour of waiting.

Also, I wouldn't say the results are nonsensical - I think it has learned a
lot more than a markov chain or a simple rnn but I agree that especially on
the surface they dont even sound like they surpass Eliza by much. Moreover, it
is significantly more apparent how much it learns about the different people
you've talked to AFTER you run it on your own data.

For a somewhat more novel/interesting result with fine-tuning GPT, I can
recommend checking out gwern's post[1] on training it on a big poetry corpus.

1\. [https://www.gwern.net/RNN-metadata#finetuning-the-
gpt-2-smal...](https://www.gwern.net/RNN-metadata#finetuning-the-gpt-2-small-
transformer-for-english-poetry-generation)

~~~
gwern
As it happens, nshepperd ran his finetuning GPT-2-small on our IRC channel.
I'd tried it before with char-RNN back in 2015 or so, and I have to say,
GPT-2-small trained way faster and better than my IRC char-RNN did.

The samples also looked a lot better than OP's. I assume that's because he ran
it for more like a day on a few hundred MB of chat logs.

~~~
gwern
I didn't run on IRC because my GPUs were busy with poetry, FWIW:
[https://www.gwern.net/RNN-metadata#finetuning-the-
gpt-2-smal...](https://www.gwern.net/RNN-metadata#finetuning-the-gpt-2-small-
transformer-for-english-poetry-generation)
[https://slatestarcodex.com/2019/03/14/gwerns-ai-generated-
po...](https://slatestarcodex.com/2019/03/14/gwerns-ai-generated-poetry/)
[http://sevensecularsermons.org/on-the-significance-of-
gwerns...](http://sevensecularsermons.org/on-the-significance-of-gwerns-poem-
generator/)

