
Fine-Tuning GPT-2 from Human Preferences - runesoerensen
https://openai.com/blog/fine-tuning-gpt-2/
======
honoredb
The story of a bug that caused the AI to optimize for maximally disturbing
text that went unchecked because the only people authorized to stop it were
asleep is a great illustration of how easy it is for an AI to "go evil" when
you're not worrying about safety.

~~~
GranPC
Link, please?

~~~
wgjordan
[https://openai.com/blog/fine-tuning-
gpt-2/#bugscanoptimizefo...](https://openai.com/blog/fine-tuning-
gpt-2/#bugscanoptimizeforbadbehavior)

(Someone didn't read the whole article..)

------
apolinario
Is there already a way for a regular person to fine-tune the 774M model in a
similar fashion as it was possible for the 124M and 355M with gpt-2-simple?

------
ayw
founder of Scale (scale.com) here! We worked with OpenAI to produce the human
preferences to power this research, and are generally very excited about it :)

~~~
gwern
Any thoughts about offering this as as service? There are lots of hobbyists
who have been playing around with GPT-2 text generation, and it'd be sweet if
you could just fire up a simple form URL with two text snippets, two options,
and it trains on feedback.

~~~
ayw
It's a good idea! They didn't demonstrate a lot of the inputs as the models
were training, but that was very entertaining of course.

------
jeffshek
In regards to improving the original dataset of 345/745M, I'm encountering
this while I've been building an open-sourced tool at
[https://writeup.ai](https://writeup.ai)

I'm not certain how random users feel when knowing their selection knowing
it's being used to improve an algorithm. In this case, it's relatively easy
for me to log the prompt and the option that was selected -- just doing it
felt a little ... bad, and I'm a little too scared of GDPR for a fun project.

The other thing is humans sometimes select the funnier option even though it
may not necessarily be the best one. In a Show Reddit post, the most upvoted
response is a Game of Throne's character Tyrion and a brothel story.

[https://www.reddit.com/r/FanFiction/comments/d5s9yh/i_made_a...](https://www.reddit.com/r/FanFiction/comments/d5s9yh/i_made_a_bot_that_writes_fan_fiction_with_you/)

Open-sourced Code: [https://github.com/jeffshek/writeup-
frontend](https://github.com/jeffshek/writeup-frontend)

~~~
Jack000
just curious how you're hosting this? The cheapest GCP GPU instance is
$0.5/hr, which is pretty expensive long-term.

~~~
jeffshek
GCP Instances that are scaling up and down based on usage. Lot of image
freezing, for faster boots, etc.

One hack is for the medium-level models, you can actually run them on Cascade
Lake (which are sort-of more optimized for ML) than traditional processors.
There's a 30% performance there. Mathematically, GPU vs CPU in performance at
inference time, you're paying an annoying premium!

Right now, the default writing style "medium" is running in Cascade Lake (no
gpu). I also (over)optimized the microservices running the endpoints too.

~~~
Jack000
didn't know about Cascade Lake, thanks! Every time I did the numbers it didn't
add up for cloud GPU. I could buy a new gaming desktop, serve the model from
that, and make my money back in 2-3 months

------
mordae
TL;DR Can you give me a summary. :-)

~~~
gwern
You can generate one yourself if you want! [https://github.com/openai/lm-
human-preferences](https://github.com/openai/lm-human-preferences)

> Note that we provide pre-trained models, so you can skip directly to RL
> fine-tuning or even to sampling from a trained policy, if desired.

