The problem is that they prevent any joke about gay individuals, not just jokes that are demeaning or derogatory. Joking about things you encounter in every day life normalizes the subject. What this treatment does is say these categories are not normal, which, I thought, was not the outcome we should be looking for.
The nuance required here is something that LLMs have not been great at, or at least not good enough to prevent actual homophobic jokes. See the general issue that LLMs struggle with negation. In fact alignment work is on some level about getting LLMs to be smart about this nuance.
I agree in the abstract that, for instance, an LLM could/should be able to make jokes about Tim Cook — just because he’s gay doesn’t mean he should be off limits for a joke about Apple, for instance. That would demonstrate an impressive level of nuance.
But also like … what’s the business play here? “Our LLM tells jokes about gay people that aren’t homophobic!” doesn’t strike me as a powerful market differentiator; or at least the market will have to evolve in very specific ways to make that meaningful.
The handwringing that I see on HN around LLMs being neutered really seems like a plea for alignment, just of a different sort: folks want the models to reflect their priors, their alignment, where nothing is off limits. The business case for this — and the general social value of this — is generally unexamined.
