
U.S. tech companies are building a Chinese-style social credit system - neuro
https://www.fastcompany.com/90394048/uh-oh-silicon-valley-is-building-a-chinese-style-social-credit-system
======
caymanjim
Your FICO score and detailed credit history already serve this purpose to a
large degree. Everyone knows that affects whether or not you can get a loan
(and thus purchase a car or home), and on the surface that seems fair enough,
but is already fairly discriminatory. But it's far worse than that; if you
have a bad credit score, you can't even _rent_ an apartment. And many
employers won't hire you. It's not just used to calculate what interest rate
you get charged; it's used to judge your value as a human being in general.

~~~
adrr
FICO score isn't used for employment and I doubt it is used from renting.
Employment based credit pulls gives employers a stripped down credit history
allowing employers to see total debt, late payments, payment history etc. As
for renting, it would be discriminatory against certain religions that forbid
debt. If you don't have debt, you can't have a credit score.

Edit: People really need to know the difference between FICO score and credit
history. You can't challenge algorithm of a FICO Score number. You can
challenge incorrect data on your credit report.

~~~
caymanjim
At least in the US, you are wrong on both accounts. Employers regularly check
credit history, and every apartment rental agency I've ever dealt with has as
well. They might not have an explicit numeric FICO threshhold that they use
(or at least admit to using), but that's just semantics. They will pull your
credit history, and if they don't like what's on it (or if you simply don't
have one), they won't deal with you.

I lived outside the US for years, and even when I'm in the US, I don't
participate in the credit market. I'm completely unable to rent apartments in
NYC unless I work out an unofficial sublet arrangement (and get treated like a
second class tenant at risk of eviction at all times), and even in the
suburbs, it severely limits my apartment choices. The only reason it hasn't
hurt me with employers (that I know of) is because I have a solid resume and
references.

~~~
turtlebits
You should be able to do what most people with not to great credit do, prepay
some amount of your total lease. Credit is to show that you pay on time. It
makes total sense you're not able to rent without some validation that the
landlord will get paid.

~~~
caymanjim
Alas it's not really like that. The NYC rental market is incredibly tight.
Most places already want a deposit plus first and last months' rent, which
really just means double the deposit. Even if you have the cash for yet
another deposit on top of that, they often won't entertain the idea. It's
easier if you can deal directly with a landlord; agencies will generally have
nothing to do with you. That and proof of a job/income should be way more than
enough proof, but it's still not.

I'm a capitalist and a bit of a left-leaning libertarian, so I'm not saying
landlords shouldn't have the right to act like this, or that I think the
government needs to step in and regulate it. If I owned a property, I wouldn't
want the government telling me whom I can rent to (more than they already do).
Just providing examples of the creeping social credit system we have. I don't
think it's about financial security once you're already asking for three
months' rent just to move in; the financial part is pretty well covered by
then.

~~~
turtlebits
The government has little to no say in who you rent it to. If I as a landlord
want 100% of rental term up front, that's up to me and has no bearing on
credit score, that is just a metric that helps me make a decision.

------
Vaslo
_" The most disturbing attribute of a social credit system is not that it’s
invasive, but that it’s extralegal. Crimes are punished outside the legal
system, which means no presumption of innocence, no legal representation, no
judge, no jury, and often no appeal. In other words, it’s an alternative legal
system where the accused have fewer rights."_

Also, what if you disagree with me politically about illegal immigration -
does that give you a "right" to ban me from your establishment? Imagine it on
the other side as well for people who are against abortion.

~~~
astura
Id imagine I wouldn't be welcome in a church or mosque because of my views on
abortion.

~~~
clarkmoody
Churches that don't welcome all aren't really functioning as churches. More
like social clubs.

    
    
      And Jesus answered them, “Those who are well have no need
      of a physician, but those who are sick. I have not come to
      call the righteous but sinners to repentance.”
    
      Luke 5:31-32

~~~
inscionent
Well functioning Churches are social clubs. Fellowship, witnessing and
counseling are all found in Christian traditions of worship.

------
mAEStro-paNDa
_Honest_ discussions and reporting around anything involving China is quite
difficult to come by these days. I would very much recommend everyone take any
Western reporting with a grain of salt, especially from obvious pro-U.S.
sources. Much of the coverage of the situation in HK can be seen as evidence
of this. On this topic specifically, Wired's recent article over this was a
breath of fresh air[1] for the overall state of news reporting when it comes
to China.

Some clarification: First, when I talk about this slant in reporting Chinese
affairs, it's not from a pro-CCP or pro-China position. Discussions about
Chinese media and bias are still essential to have, but has nothing to do with
the point I'm making here. Next, this issue isn't limited to just US media.
There is arguably an observable bias even when it comes to Western academics
that study or cover China in some capacity.

I'm sure aspects of this system in China earns a healthy dose of criticism and
skepticism. However, it's important to consider the way this may be reported
in the West, especially as tensions heat up between China and the US. Just
think, for example, that it would not be very difficult to cover the US's
credit score system as authoritarian, racist, or Orwellian. In fact, such
cases have been made in the past and have some weight to them.

Just a thought.

[1] [https://www.wired.com/story/china-social-credit-score-
system...](https://www.wired.com/story/china-social-credit-score-system/)

~~~
Aunche
> Wired's recent article over this was a breath of fresh air.

I agree that this article is pretty good. Ironically, Wired's first article
about China's social credit system generated a lot of misinformation about the
social credit system in the first place. Nothing was factually incorrect, but
most of the article was pure speculation and then everyone on the internet
treated that as fact.

------
andrerm
Why do you think Google is indexing all your online purchases?

Look, online tracking inevitably leads to behavioral profiling. Put two
behavioral profiles side by side and you have scoring, ranking. Add some
weight on behaviors, age, income etc. and you have a social credit system.

Edit: the question is not if big tech already have social scores in us or not.
The questions are when and how they will use it

~~~
word-reader
Exactly; let's say you have a social media record or even a browsing history
that indicates labor activism (or drug use, or health issues, or any other
kind of activism). That data is saved somewhere, associated with you directly
or just assembled into a behavioral profile for ad targeting purposes -- at
least for now and as far as we can tell. And like racial bias, this data can
be dumped into a neural net with a score at the end to launder what exactly is
going on.

------
dogman1233
Fully, fully agree.

Tech has opened up "channels of analysis" that the US spent a huge amount of
time, effort, and social strife legislating against. Because of the hub-and-
spoke model of tech and data, a lot of these channels have been re-opened in
indirect, but socially important ways.

What goes in one end for social media, comes out the other end in your
insurance rates. How do we think Rocket Mortgage generates am instantaneous
rate, when mortgage lenders used to rely on a great deal of relationship
management and building to do the same loan issuance (and, they had credit
scores back then, so it's not only an API into FICO that's changed).

There are so many unknown unknowns here now. Previously, your mortgage rate
bumped up if you lived in a red-line neighborhood. This was legislated
against. Now, is the same thing happening with online banking/online health
insurance rate quotes, etc., if you have a history of social media locations
that place you in minority neighborhoods? Odds are, I bet yes, or something
very similar.

The more people that realize what is going into seemingly innocuous uses of
tech - game apps, social media, food ordering, comes out in the other end in
things that really matter to us - banking (this guy's kid spends $1k a month
on candy crush), insurance (see the article), politics (we all know this), the
more this can start to be legislated safely, and at least make the consumer
aware.

------
zelon88
> That Instagram pic showing you teasing a grizzly bear at Yellowstone with a
> martini in one hand, a bucket of cheese fries in the other, and a cigarette
> in your mouth, could cost you. On the other hand, a Facebook post showing
> you doing yoga might save you money.

It will be a cold day in hell when any insurance company investigates you for
potential fraud, decides you're not guilty, and then decides to _LOWER_ your
premiums as a result of their investigation. Premiums don't work that way.
It's frustrating to see that float around as even a possibility from a blog
like FastCompany that claims to know how the business world works. I find it
incredibly frustrating anyone would even insinuate that insurance companies
have a conscience.

~~~
pfranz
Insurance companies already give away FitBits, have "health fairs," I remember
getting a $50 Visa gift card for participating in some health thing at work.
These are all in exchange for lower premiums your employer is likely
negotiated.

I've heard similar stories about car insurance companies asking to install GPS
into your car for discounts or per-mile billing.

I'm not quite sure about the end-game for insurance companies for things that
aren't about catching fraud. If we have perfect data then we lose the risk-
pooling for insurance--then you're no longer insuring against anything and
that seems like a different type of business to me.

~~~
Nasrudith
The innocent explanation is if they see it as reducing the risk in their pool
- the equivalent of it is cheaper to give away flu vaccines than say 25%
reduction in flu hospitalizations it is a win-win. Assuming it is always the
case is naive however.

However the tracking is suspicious. Insurance companies have a bad habit of
trying to never pay what they promise out of greed - despite doing so sawing
at the branch they sit upon.

As a rule of thumb if the insurance company learns nothing from what it gives
away it is probably trustworthy.

------
throwaway_law
Isn't this already done to a great extent through black balling?

SV recruiters blackball candidates/potential employees (often for no reason
than petty vindictiveness);

SV incubators/VCs use blackball lists;

even insurance has internal lists in order to assess and deny claims based on
nothing other than the "social credit" of the claimant rather than the merits
of the claim itself. They even go a step further and insurance will rate/rank
a claimant's attorney, so you may have a good claim that gets denied because
you have a low ranking attorney, or you may have a weak claim they approve
because your attorney is highly ranked (probably has a number of jury verdicts
in similar cases).

~~~
dominotw
> SV recruiters blackball candidates/potential employees (often for no reason
> than petty vindictiveness);

How do you know about this blacklist? How is it shared and maintained.

~~~
babesh
It's not a written down list. The way it works is that when you are considered
for a company, someone there might reach out to people they know at your
former or current employer and get the inside info on you. Those people are
usually higher up the hierarchy than you. So if they have some grudge they
hold against you, then you won't get hired irrespective of how you did in the
interview and you will never know the real reason.

This is the hidden power structure that crosses companies. And this power
structure holds and maintains all sorts of biases: alumni, ethnicity,
religion, gender, socioeconomic, etc... It works the other way too in terms of
some people getting hired who otherwise wouldn't.

In fact, that person holds the relationship with the other person in higher
regard than you and will in fact report to the other person that you have been
interviewing at another company. Applies to recruiters, managers, HR, you name
it.

------
eerrisfs
This makes me wonder which company made this "Social Credit Drone Strike List"
[1]

> In 2014, former CIA and NSA director Michael Hayden said in a public debate,
> “We kill people based on metadata.”

> According to multiple reports and leaks, death-by-metadata could be
> triggered, without even knowing the target’s name, if too many derogatory
> checks appear on their profile. “Armed military aged males” exhibiting
> suspicious behavior in the wrong place can become targets, as can someone
> “seen to be giving out orders.” Such mathematics-based assassinations have
> come to be known as “signature strikes.”

1\. rollingstone.com/politics/politics-features/how-to-survive-americas-kill-
list-699334/

~~~
freeflight
> Such mathematics-based assassinations have come to be known as “signature
> strikes.”

Meet the algo behind the math assassinations, its name is literally SKYNET [0]

[0] [https://arstechnica.com/information-
technology/2016/02/the-n...](https://arstechnica.com/information-
technology/2016/02/the-nsas-skynet-program-may-be-killing-thousands-of-
innocent-people/)

------
mountainofdeath
Fundamentally, for a service to not be discriminatory, it must be regulated
like a utility (e.g. telephone, electric, etc).

Using something that has the same functionality as a utility but is not
regulated as such raises some questions.

Also, profiling social media is nothing new. It's not uncommon for insurance
companies to hire private investigators to look into suspicious cases. One
case I heard was a man going on disability citing being homebound but the
private investigator found evidence completely to the contrary. I fail to see
how this is any different.

------
squirrelicus
Adding reasons why opting out of social media 10 years ago was and remains a
good idea.

------
Nasrudith
Comparing it to social credit is disingenuous even though there are real
problems with existing and proposed ratings and their applications.

Social Credit is a wrong-headed authoritarian tool of control that is enforced
and has jack and shit to do with actual creditworthiness. It only "works" from
being pressures because otherwise the companies shouldn't give a shit if they
want to optimize for profits. If you tried to sell a loan evaluation system
for banks based upon how they treated their parents in the US or Europe they
would tell you to not waste their time again.

There are problems with the current scoring systems of course - the burden of
proof for identity theft is utterly backwards, credit scores have major "how
good of a cash cow are you" aspects mixed in to what should be pure
reliability, and idiots in recruiting use it for employement evaluation when
it is utterly irrelevant.

Even if the system winds up unjust and stupid there are large differences -
the comparison isn't helpful.

------
JohnFen
Indeed so. That is one of the major reasons why I don't use social media, and
go to great lengths to evade the pervasive spying that major tech companies
have brought into fashion.

------
lidHanteyk
Didn't this happen before with some group called "Klout" or some such? I don't
remember what happened to them.

~~~
the_watcher
Purchased by Lithium Technologies in 2014, shuttered in 2018. [0]

[0] [https://en.wikipedia.org/wiki/Klout](https://en.wikipedia.org/wiki/Klout)

------
olivermarks
This could and should be a huge US election issue. Tulsi Gabbard is suing
Google and has made some pretty strong statements about big tech. I fear all
the other candidates are under the sway of the DC lobbyists where Google. FB
et al spend tens of millions

~~~
olivermarks
[https://www.bloomberg.com/news/articles/2019-01-22/google-
se...](https://www.bloomberg.com/news/articles/2019-01-22/google-
set-2018-lobbying-record-as-washington-techlash-expands)

------
mostvexing
These false equivalences are not helpful at all.

~~~
pertymcpert
Why are they false?

------
cameronbrown
In China, the government has the ability to make me disappear whereas a
silicon valley corp. can't. That's the major difference here.

~~~
Finnucane
True, they can just ruin your life, but at least they can't just kill you.
That makes it totally okay.

~~~
cameronbrown
That's at least something citizens can act upon. In China they'll just bury
you.

~~~
anamexis
Just like citizens can act up on, say, getting blackballed by Google and
having their email gone forever?

~~~
vorpalhex
Are you comparing losing your email versus being kidnapped and held in jail
without access to a lawyer or courts?

~~~
anamexis
No, I'm saying this is a thing that deeply impacts people's lives which they
cannot, in fact, "act upon" when they are wronged.

~~~
cameronbrown
Take regular backups has _always_ been good advice, that doesn't go away just
because a company handles your email for you. Always take archives from cloud
services just like you backup HDDs, use a custom domain to forward inbound
email traffic and you'll be 100% protected.

You chose the worst example of Google being able to ruin somebody's life when
if they really wanted to be evil, they'd have a monumental amount of data for
blackmail, the ability to silence you online, etc..

------
bassman9000
As long as it's not state enforced, not equivalent.

------
mytailorisrich
Insurances assessing risk based on publicly available information (social
media posts) is in no way comparable to a social credit system.

PatronScan looks potentially more dangerous, but the law in the US and UK (for
example and afaik) is that you are free to refuse service to anyone you please
as long as it is not illegal discrimination (e.g. in the UK, race or sexual
orientation).

Once again what this highlights is the power gained by these online platforms.
On the one hand as private companies they have no obligation of universal
service, on the other hand some of them have so much power that being excluded
has a real impact on people.

This reinforces my opinion that either these tech giants will effectively
rule, or they will have to be controlled in a way similar to what China does
in order to keep decisions on censorship, exclusion, and provision of service
within public hands.

~~~
enriquto
> the law in the US and UK (for example) is that you are free to refuse
> service to anyone you please as long as it is not illegal discrimination

Just wondering what does it mean and how can this ever be enforced? Does this
mean that you can refuse service to anyone as long as they are not part of any
minority?

~~~
mytailorisrich
In the UK there is a list of so-called protected characteristics that
originates from EU law. It includes sex, race, age, sexual orientation,
religion. It is not legal to discriminate based on those. In general it is
legal to otherwise refuse service. I suspect the US are similar.

On the other hand, for example in France it is illegal to refuse service
unless you have a good reason to.

So in the UK if I walk to a market stall with money in my hand they are free
to refuse to sell to me (I suspect the same is true in the US), but in France
they would need a 'good reason' to refuse.

~~~
enriquto
> So in the UK if I walk to a market stall with money in my hand they are free
> to refuse to sell to me,

But I do not understand how can this be enforced. If for example you are gay
and are refused service, can't you claim that it is due to your sexual
orientation? Conversely, if the shop wants to expel you because of illegal
reasons, can't they always claim that it is due to some other, ridiculous but
valid reason?

~~~
mytailorisrich
Yes you can obviously claim that, and yes they can obviously claim this was
not the reason.

Then it's the usual job of the courts to try to uncover the facts.

