
China’s plan to organize its society relies on ‘big data’ to rate everyone - walterbell
https://www.washingtonpost.com/world/asia_pacific/chinas-plan-to-organize-its-whole-society-around-big-data-a-rating-for-everyone/2016/10/20/1cd0dd9c-9516-11e6-ae9d-0030ac1899cd_story.html
======
truffelion
I feel like a crazy person every time I try to point out that North America is
already there. We've just done it in a less reductionist manner, in which
multiple scores are used for different purposes, because you can discriminate
better with better targeted data.

The IRS rates you on likelihood of fraud, border patrol rates you on your
likelihood of breaking residency law, the FBI rates everyone's chances of
being a terrorist, credit agencies rate your worthiness of a loan, insurance
companies rate your chances of being a cost, banks rate your likelihood of
paying a mortgage, your friends rate your worth and you let them, and if
you're in the media biz everything comes down to a numbers game. Your income
is used to sort you into pidgeonholes in virtually every aspect of your
interaction with every social institution.

Your worth as a human is already determined by a set of numbers; we've just
done it so much better than China, because nobody noticed.

~~~
woodruffw
As far as I know, none of the agencies you mentioned have _individual
numerical scores_ for 330M+ Americans and US residents. The closest thing is
credit scores (which the article mentions), which are metered and tracked by
_private companies_ under a broad set of federal regulations.

More to the point, your potential employers don't get access to your personal
life, beyond what you divulge to them and whatever information they get from
public records (sex offender lists, states with public arrest records, &c).
They don't know how "good" of a citizen you are, your mental health record, or
if you've ever gotten a speeding ticket. China's proposed system opens all of
that.

~~~
sangnoir
> The closest thing is credit scores (which the article mentions), which are
> metered and tracked by _private companies_ under a broad set of federal
> regulations.

Is the fact that private companies do the tracking supposed to make me feel
better? It doesn't; it only increases the likelihood that the information will
be sold to unknown 4th parties or auctioned at bankruptcy fire-sales.

Not even a month has gone by since the story broke of law enforcement buying
data off Twitter's firehose from a _private company_ to track protesters. See
also: parallel reconstruction.

~~~
unityByFreedom
> Is the fact that private companies do the tracking supposed to make me feel
> better?

YES!! A government has much more power than a company operating under it,
particularly in a single-party state.

------
aub3bhat
I just finished watching the Nosedive (Season 3, episode 1) from Black Mirror
on Netflix, it shows a similar system. Unlike other Black Mirror episodes its
quite well balanced. I would highly highly recommend watching it.

~~~
anexprogrammer
I watched it last night and was struck by how much impact had been lost by
Netflix transition. OK they were clearly going for humour, but even with
bigger production, and good acting the longer running time diluted it so much.
We didn't need the star rating explained every 2 mins for the first 20.

Though I think it was that one that's based on a Charlie Brooker story, rather
than written by.

I actually paused after to decide if I was going to bother with the rest at
all. 2nd episode (as far as I've got) was much better.

~~~
aerovistae
I just finished the third and have liked all three but agree with you
completely. The original runs packed far more punch than what I've seen so
far.

~~~
aerovistae
Strike that, the fourth episode is amazing and as good as any made previously.

------
11thEarlOfMar
My concern is that China will use the scoring to make moral judgments about
citizens. Yes, we have credit scores and DMV scores and some of us wind up on
white lists (PreCheck) or black lists (No-Fly) for air travel.

But determining business risk is different from determining whether someone is
a moral or immoral person, and then using that subjectivity to control the
population. It's value to the regime is in that subjectivity: If you threaten
my political aspirations, I'll score you as immoral, report your
transgressions on the nightly news and toss you in prison to be reprogrammed.

The US separates church and state to prevent this.

~~~
girzel
The problem with this happening in China is that, given the rampant
corruption, authoritarianism, lack and transparency and lack of rule of law in
China, the system is pretty much guaranteed to be expressed in the worst
possible way: used for population control and political repression, and gamed
by those in power for their own benefit.

Something really bad is happening in China. There's been talk for a few years
about abolishing the _hukou_ system of housing registration, which would
appear to be a step towards liberalization, but at the same time the
government seems to be running full tilt in the other direction, developing
new systems for control. I just left China after fifteen years, and when I
moved out of my apartment in central Beijing, an Indian guy was considering
moving in after me. The local police station told my landlord, "no Indians".
Later they told her, "no one without a Beijing hukou can live there", which is
pretty staggering if true, as the majority of Beijing residents don't have
Beijing hukou.

I don't know what's going on, but levels of paranoia in the higher reaches of
power are spiking, big time.

~~~
idra
> The problem with this happening in China is that, given the rampant
> corruption, authoritarianism, lack and transparency and lack of rule of law
> in China, the system is pretty much guaranteed to be expressed in the worst
> possible way: used for population control and political repression, and
> gamed by those in power for their own benefit.

Well said. This can be said about pretty much any "system" being introduced in
China.

------
dkarapetyan
This is what the DMV does with the point system. The FBI and CIA I think also
collect statistics and profile relevant individuals. The difference I guess is
that most of these systems are in separate silos. The Chinese government is
taking the next logical step and linking it all together. Reminds me of a
quote

 _Your scientists were so preoccupied with whether or not they could, they
didn’t stop to think if they should._

~~~
cscurmudgeon
DMV tracking whether you are driving drunk is not the same as the Chinese
govt. tracking whether you praise the govt. on Facebook.

> The FBI and CIA I think also collect statistics and profile relevant
> individuals.

Citation needed (especially that they do it for everyone).

~~~
Hondor
The US does track what you say on Facebook to see how much of a terrorist or
criminal you might be. Just because you think "praise the government" isn't
worthy of reward, doesn't mean it isn't. Afterall, China doesn't have a
democracy, so if people are too dissatisfied with the government, instead of
voting it out, they might have a revolution, which would be disastrous. In
that environment, praising the government can be helpful to keep people safe
from violence.

~~~
cscurmudgeon
I agree it makes sense for the Chinese govt. just as it made sense, for
example, for a slave trader to put down slaves that didn't toe his line and
reward those that obeyed him fully.

You know what might make more sense? The Chinese govt. (or any other similar
aspiring govt.) should spend billions on BCI research and implant chips in
everyone so that any dis-harmonious thoughts can be culled right at the
source.

------
berntb
As I understand it, the reason the world have become more democratic and
liberal for generations was because freedom made a society work better. The
open societies out competed the closed ones.

If this model really works for China and other non democracies, so they can
keep a liberal economy and still control people hard politically, then:

1984, here we come. :-(

~~~
daodedickinson
It really does seem like, with so much distraction and so little necessity,
we're nearing the end of the democratic stage of the cycle, where people use
freedom chaotically.

------
nabla9
Trackers and social media companies try to build similar scoring systems here.

If banks or insurance companies can figure out how credit score changes of
Facebook friends predicts you financial status, better abandon them if they
screw things up.

[https://mic.com/articles/123452/facebook-to-let-banks-
discri...](https://mic.com/articles/123452/facebook-to-let-banks-discriminate-
based-on-friends-credit-score)

------
walterbell
Project Cybersyn in 1970's Chile,
[http://www.newyorker.com/magazine/2014/10/13/planning-
machin...](http://www.newyorker.com/magazine/2014/10/13/planning-machine)

 _" Beer was a leading theorist of cybernetics—a discipline born of midcentury
efforts to understand the role of communication in controlling social,
biological, and technical systems. Chile’s government had a lot to control:
Allende, who took office in November of 1970, had swiftly nationalized the
country’s key industries, and he promised “worker participation” in the
planning process. Beer’s mission was to deliver a hypermodern information
system that would make this possible, and so bring socialism into the computer
age._"

~~~
idlewords
Cybersyn was about giving the government what we would call a 'dashboard' of
the planned economy (with Star Trek chairs), not social control.

------
gragas
How incredibly dystopian of you, China. This is also pretty funny considering
it is exactly the premise the first episode of the new _Black Mirror_ season.

~~~
mirimir
Well, I recall news from at least 1-2 years ago about these Chinese plans.

------
cpeterso
This reminds me of the "Whuffie" social currency from Cory Doctorow's science
fiction novel _Down and Out in the Magic Kingdom_. A person's current Whuffie
is instantly viewable to anyone, as everybody has a brain implant giving them
an interface with the Net.

[https://en.wikipedia.org/wiki/Whuffie](https://en.wikipedia.org/wiki/Whuffie)

~~~
makomk
Which, incidentally, Cory Doctorow considers to be a fundamentally terrible
idea: [http://www.locusmag.com/Perspectives/2016/03/cory-
doctorow-w...](http://www.locusmag.com/Perspectives/2016/03/cory-doctorow-
wealth-inequality-is-even-worse-in-reputation-economies/)

------
visarga
Reddit and YC also expose each action to voting and judging, and sum up users
into karma scores. We already live in such a rating based moral system. If the
reddit karma would decide a person's worth and social rights, that would be
pretty dangerous. People would self censor, be less creative, and in general
only act in a way that protects the score. It would turn into a kind of self
policing.

~~~
apatters
True, but these are privately owned communities and if you disagree with their
standards you can go join some other community or start your own. A Chinese
citizen can't exactly go start their own competitor to the Communist Party.

~~~
ubernostrum
And yet... credit scores are maintained by private companies, and are
inescapable in the US, as are their effects on your opportunities in life.

So "It's a company, not a government" is not a useful argument to make for
this kind of situation.

~~~
apatters
The credit rating industry in the US is very heavily regulated. Setting aside
whether the regulation of that industry is good or not, there is really only
one US credit scoring system and it's effectively controlled by the
government, so this is not an example of multiple, privately controlled,
competing systems at work.

The Chinese "social score" system would almost certainly be developed and
administered by private companies (prototypes and precursors have been). The
government would just set the rules. So in fact it would be similar to the US
credit system in this way, one system, dictated by the government, private
entities just handle its operation.

------
nullc
Someone mistook various dystopian science fiction books for howto guides.

------
arjie
Well, this is bad because it's the Chinese government.

But companies like Airbnb have to use proxy metrics like number of Facebook
friends. If I could be proven to be the trustworthy person I am, then a host
wouldn't need to look at all that. If I'm known to have destroyed someone's
home, then someone else on Turo knows better than to rent to me.

In practice, it's likely to turn into an eBay style "A++++ best person ever.
Let me on the bus brilliantly! 5/5" but the idea isn't terrible on its own.

~~~
eyjfndykhx
>Well, this is bad because it's the Chinese government.

Follow up question: would this be good because it was by the US government?

~~~
paulryanrogers
No! Too much power and information in one place is always bad

------
on_my_position
This is clearly a very bad direction for China in itself. Most people reading
this instinctively want a "but", and this is one that I can provide that
applies to China and Russia (though Russia is really much freer than China):

Western critiques of China and Russia are directed at the regime, but if the
regime collapses, there is no on to criticize, but people still suffer. That
is, the Western critique is not fully utilitarian: it criticizes the regime
for taking measures to ensure stability, without considering the consequences
of the regime losing control.

For reference, about 70 million Chinese died after the collapse of the Qing
dynasty (caused in part by Western intervention, by the Opium wars), and
Russian life expectancy reduced significantly during the power vacuum between
the fall of communism and the rise of Putin.

This gives some background on why both countries support their governments:
they want stability and strong central government because they know the
alternative.

If Westerners want to adopt a moral and convincing approach to human rights in
China and Russia, they need to explain how these things can exist without
causing a collapse of law and order that would be worse for the ordinary
citizen. This would require reigning in the CIA so that countries could loosen
the reigns without fear of a color revolution or Arab spring.

~~~
kristianov
With Trump and Hillary, the community on Chinese websites are increasingly
disenchanted with democracy. One high-vote answer in Zhihu, or "the Chinese
Quora", claims that this election cycle has been a "great educational
experience for Chinese Nationalism and Marxism". You'd be surprised how
popular Trump is in China, and how much they support their Government.

~~~
hilop
If Chinese love Trump, why are they disenchanted with democracy? Trump is
doing great in democracy, in the #1 or #2 spot. And I'd assume they'd like
Trump less if they were more knowledgeable of English and USA news.

~~~
li-ch
By democracy, I think he meant hillary's brand of "democracy", which is going
to win in November.

------
pamparosendo
"Imagine a world where an X government monitors everything you do, amasses
huge amounts of data on almost every interaction you make, and awards you a
single score that measures how “trustworthy” you are."

What country is he talking about?

------
rdtsc
This is wonderful. A fully gamified life. It is so dystopian...

America has the credit rating system. With much hand wringing and energy
devoted to increasing or altering the score and so on (companies advertising a
way to increase it for you, landlords claiming they'll ruin your credit rating
if you ask for a deposit refund ...).

Now imagine if the system was even more corrupt and you could pay someone
under the table to alter your score directly. Now imagine that is not just a
number for your credit worthiness, but a number for your, well general-life-
worthiness. There is no way that is not getting exploited, hacked into or
gamed. Maybe that is the ultimate goal, it lets those who control this system,
control the population?

Imagine an implied threat of getting -150 points for joining a protest. Or
maybe your relative is going to court, you imply to the judge he might get a
+50 if he lets your nephew walk. Possibilities are endless, as they say.

~~~
dkarapetyan
Isn't this the case already? The judge will let your nephew walk if you are
rich enough. You don't need to be in China for that to be the case. What is
wealth if not another point system?

~~~
rdtsc
It is just much more efficient if you can just twiddle a number in a database
instead of say having to gift them a house. Or instead of capturing and
torturing someone, just lower their "karma" so they can never get a job first.
It is about efficiency.

------
mempko
We already do this. Everyone has a credit score. Everyone gets a background
check before renting or getting a job. Let's not pretend we don't have a
rating for everyone yet...

------
throw2016
I think the moral highground to judge surveillance and human rights issues has
been lost. Pre snowden this kind of thing would cause mass hysteria, massive
moral grandstanding and call for sanctions in western populations and media.

Now chastened its more of legalese and tiny triumps based on hope rather than
scrutiny.

Just as well the sordid culture of using human rights to score moral points
and signal cultural superiority is done and the remaining vestiges of
apologism and denial should think seriously about self examination and
scrutiny lest the ground shifts even further under their feet.

------
girzel
Coincidently, an article[1] just came down my WeChat pipeline about the
population clean-up of Beijing. It's a great, classic propaganda article, full
of pictures and statistics about how crowded Beijing is, and lots of "wouldn't
it be nice if there were fewer people?" and "look how hard your government has
had to work, don't you feel sorry for us?".

Basically they're driving out those who don't have Beijing housing
registration. The propaganda is addressed to those who will enjoy the relief
after the undesirables are gone. No one speaks to the undesirables.

The one-child policy may be over, but the Chinese government's manipulation of
the lives of its citizens is alive and well.

[1]:
[https://mp.weixin.qq.com/s?__biz=MzA3Mzc3MTc0Mg==&mid=265293...](https://mp.weixin.qq.com/s?__biz=MzA3Mzc3MTc0Mg==&mid=2652931543&idx=1&sn=9f41b93d5f96260e1b76622bc24ff239&)

Edit: Here's another link (sorry, also in Chinese) from June, laying out the
bones of the policy to drive people out of the capital. Again, posed as a
great policy with no downsides.
[http://www.china.com.cn/guoqing/2016-06/17/content_38687237....](http://www.china.com.cn/guoqing/2016-06/17/content_38687237.htm)

------
Tharre
Relevant extra credits episode:
[https://youtu.be/lHcTKWiZ8sI](https://youtu.be/lHcTKWiZ8sI)

------
michaelcampbell
I'm halfway through "Weapons of Math Destruction", and this sort of thing is
really, really scary. If the amount of damage done in the US already with big
data models, with a much narrower focus, for smaller consequences, and
ostensibly "good" purposes is any indication, this will be incredibly harmful.

------
tmptmp
I feel sad about the Chinese people and people living under communist/Marxist
regimes.

Communism/Marxism is one of the worst ideologies out there (along with
intolerant religions). The western democracy is the best model the humans have
invented yet. This is not to say we must not strive to improve it. But
Communism/Marxism is not at all an improvement, it's an outright societal
cancer. But I am seeing is the increase in criticism of US (and western
democracies) by leftists and liberals. That in itself would not have been a
big deal but these leftists praise Communism/Marxism.

This is a wake-up call to the people of united states who are listening too
much to the unjust and undue criticism of the US govt for surveillance. Post-
Snowden some leftists are taking undue advantage of the atmosphere to spread
their fear mongering anti-democracy propaganda amongst US citizens.

~~~
alextooter
You are total wrong. In fact,today China is not communist/Marxist at all.China
is more Capitalism than USA. The word communist/Marxist is only mentioned when
holding big meetings.

~~~
tmptmp
>>China is more Capitalism than USA.

China is not at all capitalist as there is no free market, and you may say
it's not communist/marxist if it pleases you but it's a communist party
controlled tyranny. *

If anything, China is much, much more tyrannical and dictatorial than USA.
Tyranny and brutal dictatorships is only what communism/Marxism have produced
in large amounts. The communism/Marxism have produced very little good and
very large evil wherever they got foothold.

* A Chinese citizen cannot even move within China without getting permits from the communist politburo approved officials, forget about him/her being able to launch a business freely.

Edit: added * details

~~~
idra
Communism/Marxism =/= tyranny and brutal dictatorships. They aren't mutually
exclusive, but one doesn't imply the other. There was Marxism and tyranny in
the Soviet Union, modern China only has tyranny.

------
dealiaro
This is kind of interesting, in a "history of communism" type of way.

There’s been a sense that communism as a goal in China has been in zombie mode
for a while with the system continuing temporarily until an economic crisis or
somesuch brings it to an end.

This reminds me of the various “rationalisation” initiatives undertaken by
state communism(s) in their early days which formed such a big part of what we
think of as communism. These were (attempts at) nitty-gritty policies aimed at
achieving the goals of big talking communist slogans and utopianism.

It seems revivalist.

In some ways, this ideas kind of makes “sense’. Digitisation, social media and
other features of modern societies lend better (at least in theory, while
wearing our 99 year old state-communism hats) to this sort of societal
engineering than their 20th century equivalents.

~~~
eveningcoffee
Zombie mode communism has been already tried right in the beginning by New
Economic Policy in Soviet Union between 1922 and 1928
[https://en.wikipedia.org/wiki/New_Economic_Policy](https://en.wikipedia.org/wiki/New_Economic_Policy)

The end was one of the bloodiest crimes against humanity and one of the most
widespread genocides of the previous century.

------
Zhenya
"Imagine a world where an authoritarian government monitors everything you do,
amasses huge amounts of data on almost every interaction you make, and awards
you a single score that measures how “trustworthy” you are."

Oh, like the US.

------
matt_wulfeck
I despise this aspect of our field. On the one hand we're ushering in a new
era of learning and knowledge. On the other we're creating tools of
unprecedented power.

They're searching for enemies of their state and they will probably find them.
Let's not ignore the fact that heroes and patriot's often start out as an
"enemy".

Sarah Winchester believed she was haunted by the men, women, and children
killed by her husband's invention. What will keep us up at night in the
future?

------
gaius
The North Koreans already do this, and they did it with bureaucrats, not "big
data"
[https://en.wikipedia.org/wiki/Songbun](https://en.wikipedia.org/wiki/Songbun)

[http://www.hrnk.org/uploads/pdfs/HRNK_Songbun_Web.pdf](http://www.hrnk.org/uploads/pdfs/HRNK_Songbun_Web.pdf)
(PDF with more details)

------
brian_herman
I've seen reports of this for at least 6mo to 1yr. Is this vaporware by China
or do they have anything more than a press release at this point?

~~~
seanmcdirmid
This is something beyond the sesame credit score talked about before.

------
powerapple
It is already in place. If you owe someone some money and you don't want to
pay. He can sue you in court, and what the court does is that before you pay
back whatever you owe, you cannot buy flight tickets, cannot take high-speech
train, cannot stay in good hotels (basically any where you need to show your
ID). agencies cannot book you holidays abroad and so on.

~~~
vijucat
> cannot take high-speech train

This is one of the best Freudian slips / unintended puns that I have ever
read. Damn right he's not taking the high-speech train anywhere in China by
standing on a soapbox and complaining about Big Brother and disrupting the
harmony of society.

------
daemonk
Playing devil's advocate here. Is the outrage towards this kind of system
essentially a criticism of the technology? Specifically, can we guarantee that
the system would be secure and can we guarantee that the system is accurate
(objective) and not subject to human manipulation?

If we hypothetically assume that the system is hacker-proof enough and
accurate enough for all practical purposes, is there still an underlying moral
reason not to implement this system? I instinctively feel that it is wrong,
but I just don't know what it is.

I think we all unconsciously rate people in our social interactions. Maybe to
explicitly quantitate that rating solidifies it and makes social mobility
harder? Our fuziness in our unconscious rating of people gives leeway for
others chances to redeem themselves or not? Whereas in a system like this,
people are more likely to be stuck where they are?

------
thomasthomas
Propaganda Games: Sesame Credit

[https://www.youtube.com/watch?v=lHcTKWiZ8sI](https://www.youtube.com/watch?v=lHcTKWiZ8sI)

------
jonmc12
All western reporting about Sesame Credit has focused on privacy. Has there
been any objective analysis on projected economic impact of the system?

------
lambdacomplete
Black Mirror S03E01 minus the government. That show is incredible.

------
jakeogh
Next up: Attach the score to "basic income".

------
jakeogh
Next up: Attach it to "basic income".

------
pmlnr
We have Facebook for us to do this.

------
mrmrcoleman
Black Mirror - Season 3

------
gjolund
Lets all pretend that the US won't be doing this in 10 years.

Facebook/Google are already working with credit agencies and law enforcement.

We make China out to be the big bad guy, but all they do is follow our
example.

~~~
hilop
Credit agencies?

