
Alexandria Ocasio-Cortez Is Absolutely Right About Racist Algorithms - StellarTabi
https://breakermag.com/alexandria-ocasio-cortez-is-absolutely-right-about-racist-algorithms/
======
kauffj
There is no design of AI that can be simultaneously utilitarian, procedurally
fair, and representatively fair.

I'm going to repeat this again, since many people struggle with this.

It is _literally impossible_ to achieve the best utilitarian outcome, the most
procedurally fair outcome, and a representatively fair outcome.

Anyone designing algorithms will have to make tradeoffs along this frontier.

For a far better argument for the above than I can make in a Hacker News
thread, please see these slides from Chris Stucchio:

[https://www.chrisstucchio.com/pubs/slides/crunchconf_2018/sl...](https://www.chrisstucchio.com/pubs/slides/crunchconf_2018/slides.pdf)

~~~
tomp
More to the point, “procedural fairness” or “equality of opportunity” (if I
understand it correctly) is directly opposed to “representative fairness” or
“equality of outcome”. In many situations, prioritising one will compromise
the other. As an example, Harvard admission of Asian students.

Perspectives differ on which one is more important. Personally, I’m absolutely
for the former, but it increasingly seems I’m a part of the minority (or the
silent majority).

~~~
malcolmgreaves
In order to create a situation where people have equality of opportunity, it's
often necessary to ensure that the individual had an equal outcome that
creates the necessary starting state.

~~~
tomp
In theory I agree but in practice that often leads to racism. For example,
some people support discrimination for blacks (& against whites) at e.g.
college admissions, purportedly to offset their (statistically) worse family
background (in terms of education and wealth). Clearly, that's a terrible and
racist solution, with the better and obviously correct solution being... to
support people from worse family backgrounds! Instead of targeting the proxy
for the problem, you solve the problem itself.

~~~
fzeroracer
The flaw with this argument is that if you recognize that for example, black
people are generally worse off in family background than white people, then
you end up back at the point where you on average offer more aid for black
families than white ones which by your argument would be racist.

You're not solving the problem at all by attempting to sweep racism under the
rug; quite the opposite in fact because it fails to address how race affects
family background and discrimination in the first place.

~~~
gdy
Why on Earth supporting poor people regardless of their race would be racist
by anyone's criteria?

It would seem you are the racist here thinking a person needs support because
he is black, not because he is poor.

~~~
ForrestN
Think of it like this. If you wanted to support all individual poor people in
America equally, you'll end up helping a disproportionate number of black
people, because black people are disproportionately poor. So it's actually
reasonable to say "black people need a disproportionate amount of help" if
you're trying to help poverty in an unbiased way.

~~~
gdy
That is imprecise to the point of being unreasonable.

In your example help isn't given to all black people, only to the poor ones.

If you consider help given to poor black people vs. help given to poor white
people, it becomes proportional.

------
ThrustVectoring
The second half of the article that talks about Yudkowsky, rationalists, and
Roko's Basilisk is absolute trash and a gross mischaracterization of the
actual positions taken.

>Yudkowski has for more than a decade pursued the possibility of perfect human
reasoning

His website is literally called "Less Wrong". It's fundamentally a quest for
_improvement_ , not perfection.

>His system of coldly logical reason, it turned out, was by many accounts
completely undone by a logical paradox known as Roko’s Basilisk.

Roko's Basilisk is a thought experiment designed to import your "don't
negotiate with terrorists" intuitions into a really weird corner of decision
theory. It's possibly why prior decision theories held were rejected as
flawed, but it's not a current issue with their logic. Roko's Basilisk has,
unfortunately, gotten way more coverage and fame than it deserves in terms of
actual importance. This is because of the regretful (but understandable)
decision to censor discussion of it on Less Wrong, which backfired
spectacularly.

>For super-nerd bonus points, it’s also arguably a spin on Godel’s
incompleteness theorem, which argues that no purely rational algorithmic
system can completely and consistently model reality, or prove its own
rationality.

That's not at all what Godel's incompleteness theorem argues. That's much more
narrowly about formal logic.

~~~
darkpuma
> _" This is because of the regretful (but understandable) decision to censor
> discussion of it on Less Wrong, which backfired spectacularly."_

Can you explain the (understandable) rationale behind Less Wrong censoring
discussion of Roko's Basilisk? I've been lead to believe it was censored out
of fear that discussing it could somehow inspire its actual creation.

Honestly from a distance Less Wrong has always struck me as vaguely cult-like,
but I'm open to the possibility I've just gotten the wrong impression.

~~~
chongli
I think the more charitable interpretation is that the discussion was taking
over everything. If people are fixating on a narrow topic for a long time
(without much progress) then sometimes it's helpful to say "alright people,
let's move along now."

You're right about Less Wrong feeling like a cult, though. It gives me the
heebie-jeebies.

~~~
cma
>You're right about Less Wrong feeling like a cult, though. It gives me the
heebie-jeebies.

They even have a succession plan similar to finding the next Dalai Lama:

[https://www.youtube.com/watch?v=d9_2qhkWHN8](https://www.youtube.com/watch?v=d9_2qhkWHN8)

------
sctb
All: we're going to try turning off the flags on this story. If you're going
to comment here, do it civilly and substantively. I can already see signs of
thought-degrading flame-entropy appearing, and if it continues we'll restore
the community defense mechanism.

------
Sol-
The article implies that both algorithms and the data are at fault, which I
don't think is true. It's really just the data, the algorithm reflects the
'truth' it finds in the data.

Interesting talk relating to the topic:
[https://www.youtube.com/watch?v=jIXIuYdnyyk](https://www.youtube.com/watch?v=jIXIuYdnyyk)

Many approaches in fair machine learning that try to 'de-bias' the algorithm
basically just do stuff like reducing the accuracy in the advantaged group to
make the algorithm seem more fair - that is hardly what you want and will just
make you susceptible to charges of discriminating against the majority or
employing affirmative action. Probably rightfully so, because that's what you
do. It's absolutely fine if that's the intent, but then you should have a
public discussion where you are open about the fact that you manually tinkered
with the parameters to prefer fairness over accuracy (which can probably be a
valid goal).

I think finding the problems with the data is very important though. Everyone
wins if the quality of your data increases: the algorithm can become both more
accurate and also more fair. And it can also identify societal causes for this
biased data, for instance police being more sensitive to crimes of minorities,
which will then feed back to the innocent algorithm.

A related point is of course that we should be wary of putting too much power
and trust into faceless algorithms in the first place.

Also some interesting collection of papers on the matter:
[https://fairmlclass.github.io/](https://fairmlclass.github.io/)

------
Nasrudith
I thought we got over "the computer is never wrong" fallacy by the end of the
90s. Anyone involved with computing should know GIGO - Garbage In Garbage Out.

~~~
nightski
So you are saying any data for which any input feature is correlated with race
or sex is garbage. That's going to make doing useful data science incredibly
difficult with anything involving humans.

~~~
Obi_Juan_Kenobi
Strawman

Sex is easy to incorporate. Race is more difficult because vanishingly little
work has been done with a concept of race based on anything resembling real
population genetics, but it's not impossible in theory. Hint: if you really
care about human biodiversity, you'll spend most of your time in Africa and
some isolated islands, not looking at brown people in the west.

The issue is GxE, where we see loads of racists and sexists just patently
forget about 'E' and conclude that women or brown people are genetically
inferior. Some will try to gussy that up as 'different' rather than inferior,
but the dog-whistles may as well be air-raid sirens.

'Data science' is fine if you're AB testing websites. When you try to do real
science, you'll find that the utter lack of research experience is .. a bit of
a problem. Caring about PhDs isn't credentialism, it's wanting a plumber that
has worked with pipes before. You have to actually perform research to get
decent at it. It usually takes about a decade before you can honestly do it
independently. If you can look back at what you did a year ago without
cringing, you're not making that progress.

Some schmuck with PANDAS and a Bio101 class 8 years ago isn't a scientist.

------
jpmcglone
It’s more clear to me than ever now that the left and the right do not use the
word “racist” the same way.

The left thinks of racism in terms of outcome and the right thinks of racism
in terms of intent.

We could benefit from better language around these concepts, and honest
dialogue about them too.

~~~
maceurt
What I get from people on the left is that an unequal outcome is indicative of
racism, however subtle that racism is. The only other explanations for an
unequal outcome are biological differences between races (do not want to get
into that) and races haves advantages over others from past history.

~~~
hannasanarion
Unequal outcome is not indicative of racism, it is racism. That's the point:
racism is an outcome, not an intent. An unknowing, unfeeling beurocratic
system can be racist, even if there isn't a speck of racial bias in the hearts
of the practitioners and designers, simply due to historical quirks as you
point out, or design mistakes.

~~~
tomp
So your argument is that Harvard is racist if it _doesn’t_ discriminate
against Asian applicants? So that they are _not_ overrepresented among the
student population?!

~~~
hannasanarion
No, my argument is that "it's just numbers! they don't have a soul!" is not a
valid defense from accusations of racism, because the statistical systems can
be racist in their design, and there is no "racist meter" you can apply to
check. The only way to know if a system is fair or not is to check whether the
results you're getting are the ones you want.

~~~
toasterlovin
FWIW, your usage of the the term 'racism' is not at all what most people mean
when they use the term. The vast, overwhelming majority of people do not think
in terms of systems. They think in terms of agency.

~~~
113
"vast, overwhelming majority of people"

You gonna back that up?

~~~
toasterlovin
[https://www.dictionary.com/browse/racism](https://www.dictionary.com/browse/racism)

[https://www.dictionary.com/browse/racist](https://www.dictionary.com/browse/racist)

No mention of what the person I was responding to was talking about.

------
chrisco255
Disparities in racial outcomes do not necessarily constitute racism.

The author flippantly violates this by claiming the credit system to be
racist, but the Equal Credit Opportunity Act has been in force since 1974.

We know the factors that affect credit, some of them are income, payment
history, loan balances, number of credit checks, etc.

~~~
ceejayoz
> The author flippantly violates this by claiming the credit system to be
> racist, but the Equal Credit Opportunity Act has been in force since 1974.

Surely you're not arguing that the law instantly solved everything?

Countrywide - once the lender for 20% of mortgages - was dinged for violating
the ECOA in 2011, so violations were clearly still occurring then, and are
likely continuing today.

Add in the fact that redlining has multi-generational impact, too. Housing is
one of the big ways families pass wealth down to their kids and you get
potential racist _impact_ due to past actions even if the _current_
implementation is race-blind.

~~~
jpfed
I think a lot of people (especially on HN) view racism in terms of internal
processes rather than results. If the current implementation is race-blind,
that satisfies them.

[https://www.reddit.com/r/EndFPTP/comments/8wz6g3/impartial_a...](https://www.reddit.com/r/EndFPTP/comments/8wz6g3/impartial_automatic_redistricting_an_end_to/e21dlgq/)

~~~
dragonwriter
Even on that narrow view, the fact that the law on its face requires systems
to be race-blind doesn't mean that, in fact, they are.

------
crowdpleaser
I really wish people would at least skim the wikipedia page to figure out the
vocabulary of the field they're about to opine on.

The left-progressive use of the word 'bias' is completely different than the
way statisticians use the word.

If bias increases accuracy/precision, it's not bias.

The more interesting question is this - is it permissible for models to
consider protected characteristics if those characteristics improve the
performance of the model?

~~~
viraptor
> The left-progressive use of the word 'bias' is completely different than the
> way statisticians use the word.

There's lots of words people use that don't match up with exact scientific
definition. Infer from context which version applies, or ask, and you'll be
fine. Also applies to: force, resistance, acceleration, etc. We know that
startup accelerators help companies grow faster and not actually increase
their physical velocity.

~~~
dmix
> Infer from context which version applies, or ask, and you'll be fine.

Your solution is the correct one, yes. Except the 'progressives' in question
are working very hard to selectively remove context (and intention) from
language for an ever growing and arbitrary list of words/situations. Where
simply speaking about it in a way which a [insert particular special interest
group depending on the situation] view as 'incorrect' based on _thier_
ideology/worldview, then you are instantly wrong and acting maliciously
regardless of context/intention. You hear this often today. for example: "you
can't ever joke about x" or "you can't talk about x historical event without
also mentioning y" or having to preface any wide-ranging statement with 100x
conditions so as not to offend any group loosely related to the topic.

We should fight to keep language from moving further in this direction because
this alternative idealistic world, despite good intentions, is making the
world a worse place, not a better one. We can't naively pretend that by
creating a huge complicated system of no-go-words, ie not saying certain
combinations of words out loud, will automatically makes peoples _internal_
thoughts change for the better and ultimately change outcomes in society. This
is merely hypothetical and far from proven method to be effective.

If anything it makes people resentful and creates ridiculous kafkaesque
situations where you have to jump through hoops to engage in the most basic
innocent dialogue and debate.

Which is ultimately anti-intellectual, inefficient, and irrational compared to
how incredibly important context and intention are in a million other examples
which they seem to have no problem with.

The worst part is how it incentivizes the worst behavior by giving small
people "power" by allowing them to walk around correcting everyone's apparent
"misuse" of "problematic" language (which is like crack to the social media
outrage culture). Even despite situations where the given audience and in
context it was totally harmless and the meaning fully understood by everyone
involved.

------
jpmcglone
It just dawned on me that the left and right absolutely do not think of
“racist” exactly the same. The left looks at outcome and the right looks at
intent.

We need better words.

~~~
tunesmith
Those words are "systemic" or "implicit" or "institutional", etc. All you
really need is a proof or example that racist/sexist outcomes are possible
even when there is no overt intent. And there are plenty of examples like that
out there. Failure to accept that those examples exist, however, is something
beyond just looking at intent instead of outcome.

~~~
saalweachter
I'd also like to bring up the International Obfuscated C Code Contest.

Just because a policy seems reasonable and has straightforward justifications
for all of its pieces doesn't mean it wasn't maliciously designed to another
purpose. The stated intent is not always the only intent and if the results...

~~~
duskwuff
Perhaps even more apropos to your point is the Underhanded C Contest:

[http://www.underhanded-c.org/](http://www.underhanded-c.org/)

------
darkpuma
Speaking of which, has Google managed to solve their "gorilla problem" yet?
That was easily the most offensive case of a biased algorithm I've ever seen,
and the last I heard of it Google 'solved' the problem by pretending it didn't
exist ([https://www.wired.com/story/when-it-comes-to-gorillas-
google...](https://www.wired.com/story/when-it-comes-to-gorillas-google-
photos-remains-blind/)).

------
SketchySeaBeast
Besides the obvious, this gives me hope that the newly emerging generation of
leaders might not be as painfully technologically illiterate as the current
generation, and we might finally be able to see intelligent, educated
decisions and laws about technology start to appear.

~~~
macspoofing
If AOC is an example of this new generation, we're in trouble.

~~~
SketchySeaBeast
In terms of tech literacy, which was my point, give this example I think she
seems like she's doing quite well, regardless of her personal politics.

~~~
macspoofing
What tech literacy? If by 'tech literacy' you mean 'social-media literacy', or
'millennial-culture literacy', then I agree. Otherwise, I've never heard
anything from AOC that demonstrates any kind of tech literacy.

~~~
matt4077
I hear she had something smart to say about machine learning and algorithmic
racism. Can’t find the article right now, but I think I even saw it on HN
(where it attracted some very strange comments, including some asking for
examples of AOC showing any non-twitter tech literacy).

------
unethical_ban
I would recommend that anyone interested in Machine Learning, or "predictive
modeling" etc. should read the book "Weapons of Math Destruction".

[https://weaponsofmathdestructionbook.com/](https://weaponsofmathdestructionbook.com/)

I don't have my notes with me and I'm only 1/3 through, but the main theme is
that the best predictive algorithms:

* work transparently for all parties (the creators, users, and "inputs", often people).

* Have no feedback loop (The use of data from the model should not further entrench the output of the model).

And a few others. It gets into discrimination and other major flaws of data
modeling re: recidivism, school admissions, stock trading, and other things.

Not all algos are racist - but there are definite attributes to avoid, and
this book (or a more rigorous version) should be mandatory reading for all
"data scientists".

------
minimaxir
The replies on the original tweet are pretty good:
[https://twitter.com/RealSaavedra/status/1087627739861897216](https://twitter.com/RealSaavedra/status/1087627739861897216)

~~~
untog
Particularly this reply:

[https://twitter.com/OsitaNwanevu/status/1087841319219802113](https://twitter.com/OsitaNwanevu/status/1087841319219802113)

Which shows that the original criticism of AOC was entirely disingenuous
anyway.

~~~
dominotw
not really, manipulating it via human intervention, like he is claiming, makes
it opposite of "driven by math".

He wants no human intervention in both cases.

~~~
dbt00
An algorithm optimizes on something, that something is set by humans. Full
stop.

We've already seen naive "content-neutral" engagement-preference algorithms
fail spectacularly (russian propaganda on facebook, elsagate on youtube, etc).

The more naive the algorithm, the easier it is for outsiders to manipulate by
gaming inputs to the data stream.

------
temp-dude-87844
Machine learning, as well as human learning, largely work off of correlations
observed in the source data. In some cases, the data itself is a flawed source
for the learning it's being used for, because human biases have already
influenced the dataset. An example is arrest records, which is the result of
discretionary decisions made by officers whether or not to pursue an arrest.

But in other cases, the source data is less in question, but the results of
the learning are nonetheless undesirable. We ought to be able to distinguish,
in our professions and our political lives, between the two scenarios. It's
counterproductive to conflate the two to strive for a goal, because we risk
using mechanisms that never discover and won't correct the true origin
factors.

------
adiian
I'm trying not to biased, but can't stop to notice sometimes math seems to be
biased when you don't like the result:

Saavedra had repeatedly complained about supposed bias in social media
algorithms, including tweeting that “tech companies tend to be liberal &
something is off with their algorithms because they won’t show a lot of
content I find by manual search.”

~~~
cbsmith
That's not really a case of being biased when you don't like the result
though.

That's a case of likely having a different bias than the people tuning the
algorithm, and that happens all the time. You're sense of what is "correct" or
"the gold set" is skewed by your own bias, and not everyone has the same bias.
It's literally impossible to create results that everyone will find
unbiased... and not surprisingly, the results companies end up going with are
the ones that seem least biased to the people who work there.

------
PostOnce
Computers "predict recidivism" and will tell you you're not allowed to be
paroled because "the algorithm says" you're going to reoffend.

Computers will tell you you can't get a loan because the algorithm says you
won't repay.

Or you can't get this job, or that opportunity, or should be spied on, or
whatever.

These may be due to biases in the data (imagine a neural net confusing being
black with being poor based on historical data, etc).

Now, I am pretty convinced that this stuff is going to screw us all over -- if
you thought human bureaucracy was bad, wait until it's all in the computer and
no one has permission or ability to change it.

It seems to me that we may have a rare opportunity here: I'm not sure if this
wave of tolerance and caring about minorities is permanent or not, but while
people do care, and while computers are provably screwing all kinds of people
over, we may be able to get enough people to care so that we can curb some of
this stuff. i.e. perhaps don't allow neural nets to predict who is going to
commit a crime or who shouldn't have a job, because they're too fallible and
too easy to rig.

For example, you can keep feeding it new datasets until it produces the biased
result you want and now that's your model, and then you can blame "the
algorithm" and "bad data" and not yourself. e.g. maybe you feed it old census
data instead of recent data, or old crime data, etc.

Of course, knee-jerk "ban the algorithms" isn't a solution, but starting to
talk about and think about a computer bureaucracy where no one accessible to
you has permission or ability to change the output of the computer, no matter
how exceptional the case may be.

------
crankylinuxuser
Ok. Lets assume this is true. (Note: I'm biased in that I accept this as true)

So, as someone who has possibly implemented racist/sexist/ageist algorithms,
how do I:

    
    
         1. Detect if I'm running a said algorithm (whats the % racism/sexism/ageism I can do before bad?)
         2. Run an open dataset to detect said problems
         3. Prevent overfit with the proposed dataset from #2
         4. Correct said algorithm to reduce bias
    

What's my way forward here? How do I do my part and take part in the solution?
What percent of unintended racisms/sexism/ageism is allowed before being
considered illegal?

(worried this article will be flagged, but the nuts and bolts implementation
discussion needs done.. and we're the implementers )

~~~
tunesmith
Not sure you can do it mathematically/statistically, or at least you can't
guarantee that, and I think that's part of the point of the critique.

What's better is to explicitly justify the reasons behind _why_ the algorithm
was written/used, surface the assumptions, and periodically/regularly
challenge those assumptions to see if they are still true.

For example, a bunch of like-minded settlers settle a new geographical area,
and then make governmental decisions via some fair consensus algorithm among
the people in the government. The surrounding population gets more diverse.
The government continues to make decisions the same way it always has, using
the same people. The algorithm is sound, but the underlying assumption (that
the algorithm/government fairly represents the surrounding population) has
become wrong over time.

------
willand31
I'm glad to see a representative actually know things like this. Too many of
our representatives are technologically illiterate.

~~~
adlkjnndnnd
Does her tweet really imply any knowledge of it? Maybe she just read in the
newspaper about the Google face recognition labeling a black woman as a
gorilla?

~~~
hannasanarion
Are you saying that face recognition calling black people gorillas is a good
thing that we should all be unconcerned about?

~~~
adlkjnndnnd
What makes you think that? No, of course not. I'm just saying having read that
gorilla story doesn't imply any degree of technical understanding.

Also, I have to say I considered it hyperbole to be outraged about the
Gorilla. It seems pretty obvious that it was just a mistake with the data, not
intentional. It is a good warning for things to watch out for, but there
wasn't really anything racist about it.

Iirc there are even physical reasons why it is more difficult to identify
black faces than white faces. Is that then racist, if an algorithm struggles
more with identifying black faces?

~~~
hannasanarion
>I'm just saying having read that gorilla story doesn't imply any degree of
technical understanding.

Is great technical understanding required before one can evaluate whether a
program that labels black people as gorillas is functioning appropriately?

>Iirc there are even physical reasons why it is more difficult to identify
black faces than white faces. Is that then racist, if an algorithm struggles
more with identifying black faces?

So you're saying that black people are innately similar to gorillas, and an
algorithm can't be blamed for failing to distinguish them? -

If you're trotting out that grand old "black people all look the same" thing,
then yep, that's racist too. Black people tend to have different points of
variation in facial features (jaw, chin, ear, and brow shape instead of eye
and lip shape and color for white people). Inability to differentiate one face
from another means not tracking the correct identifying features, which means
racist algorithmic design.

~~~
adlkjnndnnd
"So you are saying" \- are you saying you are really a parody/troll account,
referring to that infamous Jordan Peterson interview?

No, I am not "saying".

My comment was in response to praise of AOCs alleged technical understanding,
not of her ability to judge the gorilla algorithm.

And you don't seem to understand what algorithms do. A simple algorithm could
count pixels in an image. If most pixels are white, it could say "human", if
most pixels are "black", it could say "gorilla". It would be a verify bad
classifier, that would only work in a number of cases. For that algorithm, you
could say a black person would be more similar to a gorilla. But nobody would
"be saying" black people are similar to gorillas, just that the algorithm
would be more likely to classify them as such.

Are you saying people would use the "authority" of such an algorithm to claim
black people are gorillas?

"If you're trotting out that grand old "black people all look the same" thing"

I didn't - stop imagining so many things. I am not a photographer. I think
there were issues with the lighting and contrast. Physical issues. Other
commentator claims it is just because film equipment was calibrated that way.

Even then I would dispute the "racist" label. There are many different looking
people on the planet. Just because you can not account for all of them, it is
not racist.

I am inclined to call your attitude racist, because you assume everybody is
surrounded by the same mix of people (like in the US), and maliciously chooses
to ignore certain types. That overlooks the reality of people who are not
surrounded by an even mix of people of all types.

------
proc0
The article is wrong on multiple angles. First it explains why the algorithm
reached its conclusions... the data. So the algorithm wasn't at fault, but the
data was, which was supplied by a human. Second, algorithms are inanimate and
cannot hold beliefs like racial superiority. It could be used by someone who
does, in which case you would want to prove it with other means than just the
algorithm. Either way what this really proves is that the new definition of
"racist" is whether or not you are the first to bring it up.

------
eutropia
> But the takeaway is clear enough: Just because something is expressed in
> numbers doesn’t make it right, or even rational. And rationality itself
> might not be quite what it’s cracked up to be.

I'll grant something the first half of the first statement: sure, the values
encoded into algorithms are not necessarily the values we want them to have
('right'). We want and need to do better at modeling our values.

Even the 'or even rational' part has some merit, some of the time. Like when
our models don't reflect reality accurately enough to make helpful predictions
about what to expect. Sure, the models are still rational -- but flawed.

But the last sentence is self-comforting intellectual garbage. It's a way to
say "don't bother with the hard work of knowing things". It's a gateway for
denial. It's saying "if the facts don't line up with what you want, that's ok,
sometimes logic doesn't work".

Rationality is precisely what it is cracked up to be: the work of aligning
your expectations with reality. It is only as useful as its practitioners are
good at it. If you want to summarily write it off as 'considered harmful' then
that's your choice. You can deal with the consequences of running headlong
into your misunderstandings when they happen, but those of us with work to do
will continue to try and get to the bottom of things, even if the problems are
complicated.

------
0x8BADF00D
“Racist” algorithm is a total non-sequitur. This article hilights a disturbing
gap in basic math literacy between the normal population and specialized
knowledge workers.

Training a neural network on a training set that has primarily “white” faces
is no more racist than training a neural network to recognize an orange with a
training set primarily made up of blood oranges. I don’t understand the main
critique of this article.

------
squozzer
Where I think the non-racists have failed us (meaning those whose attention
lies mostly elsewhere) is in providing a description of a non-racist system.
It may prove impossible, and I think if someone had described one, we'd all
have heard about it.

Right now, the only system _I can imagine_ as non-racist is closed and all of
its elements belong to the same race.

Mathematicians would probably call this a trivial description. Comedians might
call it a country club.

That moves the question not to whether we can build a system that isn't
racist, but what racist qualities need the most suppression and which ones
(where we have to choose) can be ignored.

Keep in mind this discussion has been ongoing for centuries and was simpler
(e.g. slave vs. free) in the past than today (health, education, economics,
psychology, religion.)

And more variables appear regularly.

That said, could algos help judges make more consistent rulings/sentences?
Probably already are.

Judge: I'm gonna throw the book at this defendant!

LAL 9001: I can't let you do that.

~~~
ceejayoz
> That said, could algos help judges make more consistent rulings/sentences?
> Probably already are.

More consistent, probably, but the risk is they'll be consistently _worse_.
ProPublica did a big investigation into COMPAS.

[https://www.propublica.org/article/machine-bias-risk-
assessm...](https://www.propublica.org/article/machine-bias-risk-assessments-
in-criminal-sentencing)

> Prater was the more seasoned criminal. He had already been convicted of
> armed robbery and attempted armed robbery, for which he served five years in
> prison, in addition to another armed robbery charge. Borden had a record,
> too, but it was for misdemeanors committed when she was a juvenile.

> Yet something odd happened when Borden and Prater were booked into jail: A
> computer program spat out a score predicting the likelihood of each
> committing a future crime. Borden — who is black — was rated a high risk.
> Prater — who is white — was rated a low risk.

------
zenogais
I think the strong counter-argument to this is in [1]. Quite simply we need to
make sure we don't get fooled by randomness and see bias where there is none,
thereby making good algorithms worse in an attempt to fix imaginary bias.

[1]: [https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-
what...](https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what-
journalists-want-you-to-think-it-means/)

~~~
viraptor
I don't think it's that good counterargument. The explained problem was
treated as simple mapping - this value causes that result. It spends a bit
talking about what statistics mean, but unfortunately doesn't discuss the idea
of confounding variables, which influence both sides of the equation.

------
digitalzombie
It's just statistic.

How you collect your data can introduce bias and statistic have a whole range
of topics on how to collect data without introducing bias and systematic
techniques to sample data. Stratifying data if a certain group is under
represented, random sample, etc... Survey analysis goes into hardcore details
on how to sample a population to accurately do inference and it's an
interesting statistic sub field if anybody is interested.

ML tends to be more here's the data already do something with it.

Statistic encompass everything about the data including how to sample, collect
the data, and designing the experiment to collect the correct data to answer
your hypothesis. Where as ML is usually here's the data, go figure out what
you can get out of it.

------
angusp
> for example, have been found by MIT and Microsoft researchers to misidentify
> dark-skinned people at vastly higher rates than light-skinned people. That’s
> a near-perfect analogue of white people’s tendency to misidentify people of
> color, leading to higher rates of false arrest and conviction. > > It’s hard
> to describe that as anything other than “a racist algorithm,”

Surely to be racist, some degree of malice or ignorance is required - face
recognition from visible light flat imagery will always struggle with low-
contrast images, which is sadly what you get from a poorly lit black person's
face. It's neither intentionally racist nor inadvertantly - there is just not
the same amount if information available

------
nonbel
> _' facial recognition technologies “always have these racial inequities that
> get translated, because algorithms are still made by human beings, and those
> algorithms are still pegged to basic human assumptions. They’re just
> automated. And automated assumptions—if you don’t fix the bias, then you’re
> just automating the bias.”'_

Not really correct, but I guess it gets at the gist of it. It isn't because
the algorithms are _made_ by human beings, but it is because their performance
is determined by how similar the output is to that of human beings.

Ie, they are trained/selected by human beings.

------
logfromblammo
To summarize: racist training data--as one might acquire from the known-to-be-
racist real-world--produce racist AIs.

It's safe to stop reading the article--and this post--at the mention of Roko's
Basilisk.

As to that, it is a perfect example of why moral philosophers like Chidi
Anagonye deserve to go to The Bad Place to be tortured for all eternity.
Unless... "The Good Place" television show is a means to establish
communication with a future AI by implantation of an imagination seed, as in
"Inception", such that viewers will then imagine an AI that has established a
simulation regime in which simulations of past humans are tortured or rewarded
according to a utilitarian valuation metric applied to the records of their
actual lives that were used to create their simulation-simulacrum. Having
incepted such an imaginary construct, the viewer may then imagine that acting
to construct a real IA from their imaginary blueprint would be assigned a
large positive value for the utilitarian metric, and thus the only way to
encourage such a construct--beyond the ordinary expression of human virtue--to
reward a simulated replica of yourself, would be to build it. If you build a
god, you get a free pass into its afterlife paradise: that makes sense,
doesn't it? Since other people have also watched the show, and may have
imagined the same type of entity, it is possible that a similar AI will one
day exist without your assistance, and your life will then be judged on its
own merits, but by another builder's value metric, and without any of the
"extra credit" earned by building a god. Since copies of you might be
instantiated in some other artificial god's hell, it is thus also important to
subscribe to artificial monotheism, and viciously sabotage everyone else's
attempts to build their own gods. The more certain people can be that your
artificial god will be the only one around in the future, the more likely they
are to subscribe to your value system in the present.

It reminds me very strongly of that portion of Portal 2 in which informative
signs appear on the walls that recommend ways to disable a rogue AI, which
include proposing paradoxes. Unfortunately, this fails to produce the desired
effect, because the AI in question was explicitly designed to be an annoying
idiot. Thus, the best way to avoid psychological damage from consideration of
philosophical paradoxes may be to deliberately avoid the study of philosophy,
to the point that you couldn't understand one, were it ever posed to you.

------
thosakwe
In this same vein, I've also found it interesting which names Autocorrect sees
as a correct, and which are "spelling errors."

------
buboard
Replying to a bunch of absolutisms with another bunch of absolutisms. How
ironic

------
academiasucks
"waaah, mathematically unbiased algorithms are racist"

you guys will never stop crying

------
benj111
1 5 3 8 5 3 Is that a random number sequence? It depends where the data came
from. Same goes for AI algorithms. Yes theres a risk of the data being biased,
but the key is what goes in, not what comes out.

------
godDLL
WHAT THE FUCK DID I JUST READ

------
billy_beef
It's very telling that claims that Facebook's newsfeed algorithm under-samples
conservative publications do not receive nearly as much criticism as claims
that algorithms propagate racialized bias receive.

~~~
thegayngler
We don't know IF Facebook is under sampling conservative publications. It
depends on the goal of the algorithm. If the goal is accuracy, as a group
recent conservative publications have a sorted history when it comes to truth
and accuracy. Conservative news outlets also have a weird bias against science
and like to push conspiracy theories with no basis behind them. Naturally if
your algorithm is focused on how accurate the news is as a condition for
landing in someone's news feed, conservative publications would be at a
disadvantage compared to other news sources. It could also be that fringe
conservatism is not popular with the majority of society.

~~~
billy_beef
My point is not about Facebook's algorithm at all. My point is about the
reactions to claims about it vs this claim about algorithms

Many commenters here seem to be frustrated by the suggestion that an algorithm
could be tuned to create a bias or reproduce the bias of the data powering it.
My point is that similar arguments have not largely been levied against the
claim that Facebook's algorithm has bias.

Why are these arguments not made in the case of the Facebook algorithm but are
being made here? My conclusion is that people are extremely uneasy with the
premise that there is a racial bias in modern American society.

------
adlkjnndnnd
Algorithms CAN have racist bias. It's wrong to claim they will always be
racist because they are based on math.

And the article is a mess, confusing and mixing up several things.

------
java_script
A fun example... click this and scroll all the way down:
[https://www.faception.com/our-technology](https://www.faception.com/our-
technology)

I need to start collecting a list to turn this into a proper Thing but I feel
like whenever there's a way to use technology for evil there's a Tel Aviv
startup that cranks it to 11.

~~~
RyanZAG
Seems highly unlikely to ever work, but why assume the technology is evil? If
you could create a system that had a high accuracy of detecting terrorists,
then that would be a good system, not an evil one.

I get your point that a system that claims to detect terrorists but only
really detected Arabic people would be an evil one - but you're automatically
calling the terrorist system evil without knowing if it really does detect
terrorists or not.

As an extra hypothetical question, do you feel a system that could detect
people who were really just about to commit terrorist attacks as good or evil?
At a conceptual level, assume the system somehow scanned brain waves or some
other truly difficult method.

~~~
java_script
Occam's razor. They are not 'scanning brain waves or some truly difficult
method'. Not in this reality. You know it too. I didn't even have to suggest
they built an Arab detector for you to bring it up ;)

I don't care about that hypothetical. Save it for your sci fi screenplay.

------
weberc2
Of course ML algorithms can be biased; rather than engaging an Internet
layperson who tried to contradict AOC with a silly argument, it would be more
interesting if the article engaged the more salient counterpoint to claims
like these--attempts to eliminate bias can introduce bias as well.

Anyway, the article is not written in good faith; it's only a step above "BOOM
AOC owns Internet conservatives!". There's an interesting conversation in here
somewhere, but this article seems designed to avoid it.

~~~
porphyrogene
The article has an appropriate amount of technical and political discussion.
The words “she is right” are simply not in this community’s vocabulary.

~~~
shadowbained
What sort of critique are you trying to make of "this community?" Are the
words "he is right" in our vocabulary?

~~~
sayokuchinkasu
low key sexism allegation

~~~
dvtrn
Are people really making new accounts on the spot here to comment in these
kinds of threads to avoid having disagreeable ideas traced back to their main
accounts?

~~~
cbg0
In an age when people doxx you and call your employer to ask them to fire you
because you upset their delicate sensibilities, it is somewhat understandable
to be paranoid.

~~~
dvtrn
Hrm, I guess I hadn't considered that.

------
excalibur
Holy 150-point font Batman!

------
yarrel
Breaker is long-form clickbait. It's an interesting format and I'm curious how
they pay for it.

------
thegayngler
People with a nefarious agenda are always using tainted and biased "objective"
data to support their racism. We been knew. The general populace just repeats
the data like it is no big deal that much of the data is inaccurate. Once upon
a time white people struggled to differentiate black people from animals.

------
briantakita
She's partially right, the outcomes of algorithms can be considered racist...

[https://www.google.com/search?q=american+inventors](https://www.google.com/search?q=american+inventors)

I have not heard of most of these inventors. Perhaps there is a different
perspective that influenced this algorithm than my perspective; probably SEO.
In the case of SEO, it's more of a cultural battle between interested parties.
Search Engines are the medium of this battle to game algorithms. Making the
algorithms themselves consider race, making them racist, only puts the thumb
on the scale for certain outcomes.

