This is a pretty disingenuous question. If you've spent any time in software development, you're well aware that tools of all sorts get used to do tasks for which they were never intended, and crazy unintended consequences result. There's no reason to think that it will be any different for pre-trained machine learning models for things like sentiment analysis.
Here's one example of how it could happen. Someone publishes a high-performing model to gauge the tone of a writing sample. This model includes the anti-Bob bias described above, such that the appearance of the word Bob is tantamount to including a curse word, and greatly biases the model toward negative sentiment. Because of its high overall performance, companies of all sorts incorporate this model into their workflows for things like grant applications, loan applications, online support forums, and so on. For example, they might use it to detect when someone is using their help form to send an angry rant rather than a legitimate request for support. Now, any time someone named Bob wants support, or a loan, or a grant, or whatever, there's an increased chance that their request will be flagged as an angry or abusive rant and denied simply because it contains their name, Bob.
In fact, we can remove the layer of indirection and note that some people have names that are spelled the same as a curse word, and already have similar issues with today's software, making it literally impossible for them to enter their real name into many forms. This example doesn't involve machine learning, since profanity filters are typically implemented as a pre-defined blacklist. But there's no reason to think that a sentiment analysis model would fail to pick up on the negative associations of profanity.
It is not a disingenuous question, it actually comes down to a question of ethics. Why would anyone with any sense of accuracy allow fiction to influence a decision being made about a person’s life?
The more talk about how people are building models, the more I want people to take these black boxes to court to force developers to explain how decisions are made.
Refusing to give someone a loan because someone trained a model with 50 Shades of Grey is unethical and insane.
People don’t usually look that deeply into the consequences of their choices. Is it ethical to invest in land mines? I’d say not, but when I last put money into a high interest savings account, I didn’t know if my bank had done that for me and it didn’t occur to me to ask.
Of course it's unethical and insane. But the point is that people are going to look at the performance numbers for the model, and if they look good, they're not going to ask how it was trained. So the fact that fiction writing was used to train the model will never come up in the discussion about whether to use it.
> In fact, we can remove the layer of indirection and note that some people have names that are spelled the same as a curse word, and already have similar issues with today's software, making it literally impossible for them to enter their real name into many forms.
Ironically, this is also an example of a system behavior that was driven by users' desires not to see certain things. Seen in a certain light, it bears a resemblance to the idea of filtering out certain associations because a user considers them distasteful.
Here's one example of how it could happen. Someone publishes a high-performing model to gauge the tone of a writing sample. This model includes the anti-Bob bias described above, such that the appearance of the word Bob is tantamount to including a curse word, and greatly biases the model toward negative sentiment. Because of its high overall performance, companies of all sorts incorporate this model into their workflows for things like grant applications, loan applications, online support forums, and so on. For example, they might use it to detect when someone is using their help form to send an angry rant rather than a legitimate request for support. Now, any time someone named Bob wants support, or a loan, or a grant, or whatever, there's an increased chance that their request will be flagged as an angry or abusive rant and denied simply because it contains their name, Bob.
In fact, we can remove the layer of indirection and note that some people have names that are spelled the same as a curse word, and already have similar issues with today's software, making it literally impossible for them to enter their real name into many forms. This example doesn't involve machine learning, since profanity filters are typically implemented as a pre-defined blacklist. But there's no reason to think that a sentiment analysis model would fail to pick up on the negative associations of profanity.