So some of these associations simply reflect the way-the-world-was or the way-the-world-is - like associating "woman" with "housewife". That's a whole debate in itself.
But some of these can be accidental. Suppose a runaway success novel/tv/film franchise has "Bob" as the evil bad guy. Reams of fanfictions are written with "Bob" doing horrible things. People endlessly talk about how bad "Bob" is on twitter. Even the New York times writes about Bob latest depredations, when he plays off current events.
Your name is Bob. Suddenly all the AI's in the world associate your name with evil, death, killing, lying, stealing, fraud, and incest. AI's silently, slightly ding your essays, loan applications, uber driver applications, and everything you write online. And no one believes it's really happening. Or the powers that be think it's just a little accidental damage because the AI overall is still, overall doing a great job of sentiment analysis and fraud detection.
The only solution I can see is forcing any company that imposes life-defining actions on people (credit bureaus, banks, parole boards, personnel offices, etc) to use only rules based on objective criteria and to prohibit systems based on a "lasagna" of ad-hoc data like present day AI systems. Indeed, if one looks at these in the light of day, one would have to describe such system as fundamentally evil, the definition of "playing games with people's lives." (just look at the racist parole-granting software, etc).
That is probably the exact opposite of what you really want. If the problem is that someone's name is Bob and the AI thinks Bobs are evil, what you want is for there to be 100,000 other factors for Bob to show the system that it isn't so. As many factors as possible, so that the one it gets wrong will have a very low weight.
Even the objective criteria will have biases. There is a significant racial disparity in prior criminal convictions, income, credit history and nearly every other "objective" factor. The more factors you bring in, the more opportunities someone in a given demographic has to prove they still deserve a chance.
You don't understand. My point is that institutions making such decision should not be able to make decisions according to these 100,000 unexplained factors. If you're a lender, you can look at employment history, records of payment and other objective related criteria. You can't look at, say, eye color, however useful you might think it is. Institutions should be able make these decisions arbitrarily, at the level that they effect lives. There should legal provisions for auditing these things (as there are, on occasions, provisions of auditing affirmative action, environmental protection behaviors, insurance decisions, etc).
But how does that help anything? The objective factors have the same potential for bias as the seemingly irrelevant ones. All you get by excluding factors is to increase bias by not considering information that could mitigate the bias in the factors you are considering.
Suppose that 80% of black men would be rejected for a loan based on your preferred set of objective factors. Of that 80%, many would actually repay the loan, but none of the objective factors can distinguish them from those who wouldn't, and when averaged together the outcome is to refuse the loan. If you used some seemingly arbitrary factors that happen to correlate for unknown reasons, you could profitably make loans to 60% of them instead of 20%.
How is it helping anyone to not do that?
People's live will depend on the decisions of these machines so people will start trying to game them. They will make sure to always purchase odd number of bananas, they will wear hats but only on Thursdays etc etc.
Now two things happen. As more people game the system the rules need to be updated. Suddenly it's all about buying bananas divisible by three and wearing hats on weekends. The people who tried to follow the previous advice got screwed over, and what's more they have nothing to show for it. Instead of making people do useful things like paying bills on time and saving up some money, it made them follow some weird algorithmic fashion. Because of this expenditure of energy on meaningless things we may see that now only 18% of people would manage to pay back loans on time.
But that's just another reason to use 100,000 factors instead of twelve. If someone's income is a huge factor in everything, people will spend more than the optimal amount of time working (instead of tending to their family or doing community service etc.), or choose to be high paid sleazy ambulance chasers instead of low paid teachers, because the algorithm makes the returns disproportionate to the effort.
If buying an odd number of bananas is a factor but the effort of learning that it's a factor and then following it is larger than the expected benefit from changing one factor out of thousands, they won't do any of it.
Given a choice between observable, identifiable and modifiable rules or hidden, poorly understood rules integral to a whole model, I'll take the former every time.
Bias will continue to exist for now. What we need to do is make sure we always build processes to appeal and review our systems, preferably in a public way.
What you touched upon is the accuracy/ bias trade-off. To have evidence in particular case you need to attempt to debias the particular system and see how it affects accuracy. Sometimes, it may even vastly improve it.
What is more important is that the systems are not benchmarked properly. As in compared against very simple metrics and systems. Such as: against random decision. Against simple recidivism prevention (grudge system). Against plain math metrics with constants.
To add, they're opaque and it is impossible to easily extract the factors that went into any given single decision. This means they act fully irrationally. Intelligently but irrationally.
Doubling down based on historical and backward looking data does not seem like the way forward and can only perpetuate entrenched bias.
All the inferences and correlations will reflect that. This is not intelligence and can only take you backwards.
No, it really isn't. In an ideal world, the reasons behind a decision are transparent, auditable, understandable, and appealable. Machine learning is none of those.
It seems like the answer to that question is situationally dependent.
In other words, by offering Adolf's below-market rates, you're exploiting a market inefficiency at no additional risk. This is an ideal world as you describe it. It's capitalism at it's finest!
An AI that was even slightly good at it would be far ahead of what we get when humans run things.
Even the greatest physicists and mathematicians can't tell the difference between correlation and causation. When people say "causation is not correlation", what they truly mean is "spurious, context-sensitive correlation is not strong universal correlation"
As a simple example:
If A causes a change in both B and C, but the change in B happens more quickly, "temporal correlation" would imply that B causes C, when that's not the case.
This is especially obvious with cyclical phenomena. The tide going out does not cause the sun to rise, for example, even though I'm sure I could draw a temporal correlation between them. Nor does my car being in my garage cause me to go to work.
If we believe certain attributes to be irrelevant to a situation, we need to build systems that are completely blind to these attributes. This is how double-blind trials, or scientific peer review works.
That's just not true. You can have systematic errors caused by bad training data which can be fixed without an increase in false negatives (otherwise new ML systems would never improve over old ones!)
In the physical world a good analogy is crash testing of cars. For decades (until 2011!!), crash test dummies were all based on average sized American males. That led to hugely increased risks for female passengers:
the female dummy in the front passenger seat registered a 20 to 40 percent risk of being killed or seriously injured, according to the test data. The average for that class of vehicle is 15 percent.
Fixing that problem didn't case any increase in accident risk for men.
It's the same in machine learning.
There is no free lunch
This isn't what the no free lunch theorem says. That says that all optimization methods are theoretically equivalent.
It's a tough problem. I think being aware that biases exist in ML is a good first step.
There is a possible causal link with names which goes beyond "children are being treated worse because of their name".
Your solution will not work, ever. All such companies will adjust their processes to maximise their own benefits, irrespective on any legal consequences. For most, the profits (in whatever way they define profit) will be the more important thing than any legal requirements placed on them. It is the nature of the people running these companies.
If you are an outlier in the data, you will remain an outlier in the system.
This is terrifying.
Do you ever see anybody change their opinion? Usually they won't change simply after hearing an argument. They change only when they have an utterly different life experience. In other words, only when they are conditioned by new data, in no ways different from how an AI changes its opinion.
I know my reaction is more to the surname than the first name.
Joseph has a solid anchor as a biblical name, though, and Stalin wasn't condemned nearly as much as Hitler was.
Do you think we should train AI to systematically ignore those sentiments?
For example, people from X race might be more likely to commit crimes in the absence of any other information (marginal probability). However, person from X race is no more likely to commit crimes than a person from Y race, conditioned on something else like where they went to school, what they do for a living, etc..
It's important to remember that AI doesn't have context, and just because it's using "data" to make decisions doesn't mean the decisions are unbiased - the underlying data may be biased.
You're not allowed to discriminate on race in housing. But is an AI that determines your creditworthiness for mortgages allowed to discriminate on what you eat, where you go to church, what your favorite music is, etc.? Maybe it doesn't have that data, but it will have one level higher - what store credit cards you have and how much you use them and where you opened them.
If you train an AI on a segregated city with a history of actively discriminatory citizens, where the few people of race X who moved into a not-race-X neighborhood got harassed out and sold before they paid off their mortgages, how easily will the AI conclude that people born in certain neighborhoods are more likely to pay off their mortgage if they avoid certain other neighborhoods?
Is that illegal? (My guess is there's no way to prove to a court, to the court's usual standards, that the AI happened to learn the city's racial tensions.) Should it be illegal, if outright racial discrimination is illegal?
This seems to be the case in London, too, so it's a bigger issue than just the US.
But for things like drug possession, yes, blacks probably get stopped much more and not let off with warnings much less.
It is like with plagues. If you put many people with plague in one area do expect that others will catch it. If the rule to segregate is silly enough do expect a correlation of plague with certain characteristics of people put together and the locations. Or their sizes. Maybe it is "cities" or "presence of slums" not skin colour, combined with overrepresentation of people with certain skin colour in them.
It takes some write genius analysis and experimentation to untangle such complex effects from causes.
Call again when you have an AI that can deal with this. Essentially a researcher AI.
If lack of money causes homicides, we should expect to see all poor rural areas have high homicide rates, too. Maybe that's true.
Another factor seems to be sex. Males commit so much more violence, so perhaps that's the only thing to focus on.
Much as there is much hype about AI and machine learning technologies, all of such systems will be, for the foreseeable future, very simplistic models of what we think we know of intelligence.
As humans, we have blindly forgotten that we know very little about the world around us, including ourselves. We think that we have a handle on reality, but we are just plain ignorant. All the models that we have for deep learning and AI or GAI are barely scratching the surface of what "intelligence" is and means.
We may get some useful tools as a result of the research being undertaken today and what we have undertaken over the last lot of decades. But we have millenia to go before we even scratch the surface of what we understand of the universe around us, let alone understand what intelligence means.
That is reasoning. I’ll agree with your later point that we don’t really know what intelligence is yet, but that’s because reasoning is clearly only one type of intelligence. Current systems are terrible at language, for example (a year ago I would have said “and spatial awareness”, but this is progressing fast and I am no longer sure).
> They cannot recognise in any way that the rules by which they operate or the data they are supplied with are or are not reasonable.
Quite a lot of humans fit that description. For example, consider how much angry disagreement the following questions get: “is climate change real?”, “Brexit, yes or no?”, and “does god exist?” Also consider how many people (angrily!) refuse to believe these questions result in any angry arguments.
But your comparison is inapt: yes, we should train AI to systematically ignore negative sentiments it associates with people named "Adolf" or "Hitler" who are not in fact the Adolf Hitler who died in Berlin in 1945. Humans have difficulty in doing so, of course, but this is a bug in human cognition which is essentially the vulnerability that bigotry takes hold of: a justifiable negative impression of one person as an individual is imputed to other people who seem superficially similar. We see one person of a minority breach a social norm or even a law, and we think that others of that minority must be prone to doing similar, simply because their minority status is salient. We fail to realize subconsciously that membership in the same minority is not actually meaningfully correlated with this behavior, and when we see someone from the majority do the exact same thing, their majority status is less salient, and we don't impute the negative impression to the majority.
A good AI should be able to distinguish dictator Adolf Hitler from, say, saxophone inventor Adolph Sax or chemist William Patrick Hitler (the nephew of Adolf Hitler), and not cast aspersions on the latter two - even though human biases forced William Patrick to change his last name to Stuart-Houston. It should even be able to understand that Indian politician Adolf Lu Hitler Marak is a separate person who merely had parents with questionable taste, and the man is not on account of his name more likely to become a genocidal dictator than any of his political rivals.
And since our justifiable negative association with the Nazi leader is, fundamentally, that he weaponized this vulnerability in human cognition, it is one way of acting on our dislike for this Hitler to make sure that the AIs we build are not susceptible to the same vulnerability.
Isn't that statement in itself an invocation of Godwin's Law?
Godwin himself wrote a little bit about it: https://web.archive.org/web/20170209163428/https://www.washi...
(And I am totally open to criticism that this particular comparison is inapt.)
You can ascribe it to "Godwin's law" as much as you like, I just find it a more realistic example than some hypothetically disadvantaged "Bob".
We should make sure that an AI, who is probably making decisions on things like legal documents / public records and not just the middle name someone goes by, will not consider it a negative that someone is named "Adolf" if they aren't Adolf Hitler specifically. And we should for the same reason make sure that an AI will not consider it a negative that someone is named "Bob" if it has a newly-acquired specific negative impression of some other person named Bob. There isn't a difference in the cases.
"Hitler" shouldn't be a special case because the rise of Hitler wasn't as much of a one-time event as we'd like to believe. When the next genocidal dictator with a somewhat rare first name gains control of a country, there will be people from the victim population who share that first name, and they should not suffer the same indignity at the hands of an AI, either. And on the flip side, when this genocidal dictator rises to power, the AI shouldn't be taught that Hitler was the only evil man who ever lived or will live; if it has the data to conclude that some actual individual (not a name) is as bad as Hitler, it should be able to conclude that.
If you are referring to the actual well-known Adolf Hitler, then there should be no need to use the name for the purposes of making decisions. You shouldn't need to use a person's name as a proxy for whether or not they are a genocidal dictator, just make the decision based on whether or not they actually are a genocidal dictator.
Or DevOps Borat: Devops is screwing things up at web-scale.
Eventually, we will have AIs which are better than us in every respect. They will embody our idea of perfection. This is stupendously dangerous. Humanity has a long history of adapting to technology 'taking away' things that everyone thought were 'fundamentally human', like the ability to do work or make things or lay train tracks or whatnot.... it's not hopeful. We deal very poorly with this.
Consider the story of John Henry. He's a folk hero. For killing himself. Because he killed himself in defiance of the machine outperforming him. So this stuff is all-caps Important. What is the likely response from humanity when there is a perfect, not machine but mind? My bet? Humanity will identify with its worse aspects. It will enshrine hatred, irrationality, mean-spirited spite, violence, self-destruction, and all of the things we built AIs to never stray into. Those will become "what it means to be human."
AI may be a philosophical crisis unlike anything humanity has ever faced. Not in kind, but simply in degree.
They're from the same set of names as Bob (and Alice), but are actually affected by this issue, although to a less extent than what you describe.
It's not certain how likely the kind of texts that uses them are to be used as training data though.
Here's one example of how it could happen. Someone publishes a high-performing model to gauge the tone of a writing sample. This model includes the anti-Bob bias described above, such that the appearance of the word Bob is tantamount to including a curse word, and greatly biases the model toward negative sentiment. Because of its high overall performance, companies of all sorts incorporate this model into their workflows for things like grant applications, loan applications, online support forums, and so on. For example, they might use it to detect when someone is using their help form to send an angry rant rather than a legitimate request for support. Now, any time someone named Bob wants support, or a loan, or a grant, or whatever, there's an increased chance that their request will be flagged as an angry or abusive rant and denied simply because it contains their name, Bob.
In fact, we can remove the layer of indirection and note that some people have names that are spelled the same as a curse word, and already have similar issues with today's software, making it literally impossible for them to enter their real name into many forms. This example doesn't involve machine learning, since profanity filters are typically implemented as a pre-defined blacklist. But there's no reason to think that a sentiment analysis model would fail to pick up on the negative associations of profanity.
The more talk about how people are building models, the more I want people to take these black boxes to court to force developers to explain how decisions are made.
Refusing to give someone a loan because someone trained a model with 50 Shades of Grey is unethical and insane.
Just take a look at people called "Null" then multiply the problem thousand times across various systems with no central appeal.
Ironically, this is also an example of a system behavior that was driven by users' desires not to see certain things. Seen in a certain light, it bears a resemblance to the idea of filtering out certain associations because a user considers them distasteful.