How did OpenAI train ChatGPT to be so (cloyingly) wholesome?

[logicallee]

For anyone who's interacted at all with ChatGPT you will notice it always has a very wholesome message. Here someone asked it to write about rotten milk:

https://old.reddit.com/r/ChatGPT/comments/ztpqsw/didnt_expect_to_cry_at_a_poem_about_rotten_milk/

and it ends on the note "friendships, love and trust can too, be forgot / So let us be careful, in all that we do / To not let the milk of life turn rotten too."

This is the end of a poem about rotten milk.

How did OpenAI train ChatGPT to be so cloyingly wholesome in all its responses, always bringing it back to a positive message?

[skilled]

Reinforcement learning combined with real humans adjusting the model's output. It's not just a bunch of text they gathered being mashed together by AI. There's an extremely strong human touch to this model.

[logicallee → skilled]

So it was rewarded/reinforced every time it brought things back to a positive message at the end?

[skilled → logicallee]

Not necessarily trained only to be positive but yes your assumption is right. I don’t know the exact details but I think there have been some OpenAI employees talking about this on their personal blogs.

If I can find the links over the weekend I will make another reply.

[logicallee → skilled]

Thanks, I haven't seen any of their blogs and would love a link.