
Goldman faces probe after entrepreneur slams Apple Card algorithm in tweets - shakes
https://www.reuters.com/article/us-goldman-sachs-probe/goldman-faces-probe-after-entrepreneur-slams-apple-card-algorithm-in-tweets-idUSKBN1XK00L
======
felideon
> David Heinemeier Hansson railed against the Apple Card for giving him 20
> times the credit limit that his wife got

Heh, _railed_. Good on DHH for using his platform and bringing light to an
issue like this.

Original thread on Twitter:
[https://twitter.com/dhh/status/1192540900393705474](https://twitter.com/dhh/status/1192540900393705474)

~~~
ryandrake
Went to the article to find all the other evidence, and no it pretty much
boils down to your single quote. Some guy is not happy with his wife’s credit
limit and went on a tweet-rant accusing sexism.

No idea how credit limits are calculated but maybe his credit limit is higher
not because he’s a man but because he’s an entrepreneur?

~~~
felideon
Sure. Someone on Twitter pointed out that GS is accustomed to catering to
business accounts. (The article here also says it is GS' first credit card.)

They, and presumably Apple moreso, should have considered this before using
the biased algorithm for their first consumer credit card. They should have
analyzed -who- would be affected by it.

~~~
FDSGSG
> should have considered this before using the biased algorithm for their
> first consumer credit card

Why would such bias be bad when assigning a credit limit? Are you suggesting
that this just isn't related to creditworthiness?

------
fabian2k
It's pretty much impossible to determine from the outside whether the
algorithm discriminates against woman (without a reasonably large and actually
random sample of customers). But that is also part of the complaint, as the
algorithm is a black box that even the customer support employees can't look
into. Humans just blindly following a black box algorithm without the option
to manually handle cases where the algorithm produces weird results has quite
a dystopian feel to me. Especially if it's used for more consequential cases
than credit card limits.

~~~
felideon
Hence the probe by DFS.

------
heavymark
Confused. They are saying that nearly all married couples (male:female) with
joint everything and the same credit history are showing consistently the
husbands having more credit? Or just one single person out of millions? As one
would assume there are probably many similar cases but where the wife got more
credit no?

~~~
tom_mellior
When I read the Twitter thread yesterday there were plenty of people posting
that they had the same observation: men getting more credit than their wives.
I don't recall seeing anyone posting the opposite.

------
ibero
there are a lot of variables that go into determining a credit score.

in this particular case, i could imagine that in his role as partner as
Basecamp there is a longer "credit trail" attributed to him just from a
business operational standpoint beyond his joint family tax filing. that alone
could explain the discrepancy between him and his wife beyond sexism.

~~~
felideon
Explaining the sexism away does not make it less sexist or defensible. The
point is the algorithm has a sexist bias, as multiple anecdotes show.

~~~
buzzkillington
>The point is the algorithm has a sexist bias, as multiple anecdotes show.

By definition anecdotes prove nothing. Until there is statistically
significant evidence of sexism this is just hearsay and your post gets close
to slander.

In this case it will be absolutely trivial to check. Just feed in the same
data for the same person only with the gender flipped.

~~~
bjtitus
The whole point of the original tweet was to point potential systemic gender
bias in the algorithm. The argument here is that the black box algorithm may
produce sexist results whether gender is included or not.

While there may only be anecdotes at this point, there should be some
investigation and discussion into whether bias may occur toward particular
classes outside of explicit gender checks. Seems like that would be done by
comparing the allowed limits with gender, race, etc. and considering why the
algorithm may unnecessarily limit those classes on a broad scale.

------
FDSGSG
Honestly it just sounds like this guy hasn't applied for very many credit
cards :)

Sometimes you get a good limit, sometimes you don't.

------
richard_todd
When I was looking through the twitter replies last night, multiple people
claimed the opposite happened (the female got the higher limit). It’s sad that
the journalist either didn’t look into it or preferred not to stray from the
implication that the algorithm is sexist rather than just buggy.

~~~
tom_mellior
When I was looking through the thread I didn't see this. Do you have links to
tweets?

~~~
ryanlol
Here's what I could find in 5 minutes:

[https://twitter.com/Flkittie/status/1193725782788628480](https://twitter.com/Flkittie/status/1193725782788628480)

[https://twitter.com/HailMaryHumble/status/119385501173590835...](https://twitter.com/HailMaryHumble/status/1193855011735908352)

[https://twitter.com/mindnotnull/status/1193794029189767168](https://twitter.com/mindnotnull/status/1193794029189767168)

[https://twitter.com/Chaubin123/status/1193729943701794823](https://twitter.com/Chaubin123/status/1193729943701794823)

[https://twitter.com/StrosniderDave/status/119369802228851098...](https://twitter.com/StrosniderDave/status/1193698022288510981)

[https://twitter.com/ksgant/status/1193666460561879042](https://twitter.com/ksgant/status/1193666460561879042)

[https://twitter.com/fauxgressive/status/1193635810475683842](https://twitter.com/fauxgressive/status/1193635810475683842)

[https://twitter.com/snitstwits/status/1193626585821319168](https://twitter.com/snitstwits/status/1193626585821319168)

[https://twitter.com/chrishedlund/status/1193539000176455680](https://twitter.com/chrishedlund/status/1193539000176455680)

[https://twitter.com/Nukes4A/status/1193341783180283906](https://twitter.com/Nukes4A/status/1193341783180283906)

[https://twitter.com/Wayner83/status/1193332092291239936](https://twitter.com/Wayner83/status/1193332092291239936)

[https://twitter.com/JeffBessling/status/1193331529453363202](https://twitter.com/JeffBessling/status/1193331529453363202)

[https://twitter.com/bdruth/status/1193315200147054593](https://twitter.com/bdruth/status/1193315200147054593)

[https://twitter.com/Dcorbett818/status/1193282534622158848](https://twitter.com/Dcorbett818/status/1193282534622158848)

[https://twitter.com/NatalieEvansMD/status/119326982416994713...](https://twitter.com/NatalieEvansMD/status/1193269824169947136)

[https://twitter.com/jetkatphoto/status/1193211571222142976](https://twitter.com/jetkatphoto/status/1193211571222142976)

[https://twitter.com/mbfuchik/status/1193143158504349697](https://twitter.com/mbfuchik/status/1193143158504349697)

[https://twitter.com/allenjs/status/1192959576477298688](https://twitter.com/allenjs/status/1192959576477298688)

[https://twitter.com/bstahlhood/status/1192816047117746178](https://twitter.com/bstahlhood/status/1192816047117746178)

Of course, I'm just some "mansplaining" asshole. You should just ignore me.

~~~
tom_mellior
Thanks for these links.

------
23B1
What's the sample size here?

~~~
arcturus17
Enough to know a 20-to-1 difference is abhorrent, given they file joint tax
returns and she has a better credit score?

~~~
23B1
Oh don't get me wrong, I'd love to understand the algorithm that determines
who gets how much credit.

But we're assuming that it has to do with gender, and none of us actually know
the difference between these two people historically. My wife has better
credit than I do but I can borrow more because my income is much higher, for
example.

~~~
tom_mellior
I think the idea behind joint accounts and joint filing is that, for all
intents and purposes, the couple have equal incomes.

~~~
23B1
But if I'm doing an actuarial and know, for instance, that X% of marriages end
in divorce, that's probably a calculation I'm going to make.

~~~
tom_mellior
If you're doing that calculation and conclude that she is somewhat likely to
end up divorced and with less income, _because she is a woman_ , then...
congratulations, your algorithm fulfills the definition of "sexism".

And just to be clear: It is not sexist to point out that women make less money
on average than men. It is sexist to treat individual women worse than men
because of this general expectation, without factoring in any other
information.

