Hacker News new | past | comments | ask | show | jobs | submit login
China’s plan to organize its society relies on ‘big data’ to rate everyone (washingtonpost.com)
204 points by walterbell on Oct 23, 2016 | hide | past | web | favorite | 163 comments

I feel like a crazy person every time I try to point out that North America is already there. We've just done it in a less reductionist manner, in which multiple scores are used for different purposes, because you can discriminate better with better targeted data.

The IRS rates you on likelihood of fraud, border patrol rates you on your likelihood of breaking residency law, the FBI rates everyone's chances of being a terrorist, credit agencies rate your worthiness of a loan, insurance companies rate your chances of being a cost, banks rate your likelihood of paying a mortgage, your friends rate your worth and you let them, and if you're in the media biz everything comes down to a numbers game. Your income is used to sort you into pidgeonholes in virtually every aspect of your interaction with every social institution.

Your worth as a human is already determined by a set of numbers; we've just done it so much better than China, because nobody noticed.

As far as I know, none of the agencies you mentioned have individual numerical scores for 330M+ Americans and US residents. The closest thing is credit scores (which the article mentions), which are metered and tracked by private companies under a broad set of federal regulations.

More to the point, your potential employers don't get access to your personal life, beyond what you divulge to them and whatever information they get from public records (sex offender lists, states with public arrest records, &c). They don't know how "good" of a citizen you are, your mental health record, or if you've ever gotten a speeding ticket. China's proposed system opens all of that.

> The closest thing is credit scores (which the article mentions), which are metered and tracked by private companies under a broad set of federal regulations.

Is the fact that private companies do the tracking supposed to make me feel better? It doesn't; it only increases the likelihood that the information will be sold to unknown 4th parties or auctioned at bankruptcy fire-sales.

Not even a month has gone by since the story broke of law enforcement buying data off Twitter's firehose from a private company to track protesters. See also: parallel reconstruction.

> Is the fact that private companies do the tracking supposed to make me feel better?

YES!! A government has much more power than a company operating under it, particularly in a single-party state.


The US may have many real privacy violations but you have at least an ostensible right to privacy.

That authority figures don't have to pretend fair even if they aren't actually fair is worth something. That surveillance isn't official is worth something. Sure, solid guarantees to privacy would be better but one really shouldn't claim unofficial surveillance is the same as official surveillance since the move from one to the other needs to be resisted also.

This is very naive. If working in many countries and for many entities all over the world though me anything, it's that:

- people are tracked by everyone. Companies. State agencies. Other people.

- the database contains much more data than you think it does. And there are more data base than you think, also most of them are not connected to each others. But people in charge are.

- the data can be exploited in much more ways than you think it can.

- the law is broken is many more ways than you think it is. And as often as it's possible when not getting caught.

- the human in charge bypass everything they can, with every way they have, and help each others to do saw across entities.

- even the human that are not willing to do any of these are not necessarily competent enough to prevent abuses.

The only we don't suffer too much about it is:

- you can't see the many things in your life that are influenced by it because it's indirect and untold. - the sheer mass of it make it hard for the human to handle everything so only 5% of the real potential is actually in used. Wait for the AI to step in... - those are life threatening most of the time. But remember the table of power and freedom always turned during the history of humanity. Our current situation is NOT permanent.

Luckily most people I've met related to the data collection are not evil enough, and don't have enough greed to be a threat. Even the ones that are are usually not competent enough to become the next big brother. But the potential is here nevertheless.

TL;DR: the amount of data is huge. It can be used in scary ways. It is abused on a daily basis, but the consequences are not too insufferable yet.

> And there are more data base than you think, also most of them are not connected to each others.

I've thought about this issue a lot (I'm a data-herding obsessed guy myself), and I think that this is one of our last chances at decent privacy, I mean, the fact that there are still many unconnected databases containing our data instead of one big blob where search for relevant info would be much easier. I'm still hoping that enough things get lost in "translation" between these several databases so that the info becomes less intrusive, and that some of the databases will eventually become "deprecated" i.e. data will become stalled.

> It is abused on a daily basis, but the consequences are not too insufferable yet.

Just the other day I found out about a guy whose credit application had been rejected by the bank because he had gotten a police ticket a couple of years ago for "public disturbance" (some neighbors called the cops when he was throwing a party at his house). What was crazy is that the matter hadn't gone through any legal court, it was just saved as a ticket stored in the police's database and probably the local tax-office's DB (the police tickets are collected by the local tax office). I can't really fathom how a private entity (the bank) got hold of that information. (I live in Eastern Europe, if it matters)

Good point. I believe also this will become a new 'standard' in most countries across the globe in this century. And this order is inevitable - the only hope that the humanity will move on, after living a few generations under this order. The consequences are too invisible for people masses that it's impossible to resist it now.

> China's proposed system opens all of that.

Don't you think it can be a big threat to national security. And why does Chinese government do that?

Any bureaucratic system will assume the features of its creator - for North America that would mean collusion between the public and private sectors. I guess that in eastern countries you will get something uncompromisingly cruel on the one side - but it will also have its loopholes that can be gamed/hacked by the subjects/citizens.

ah, i forgot to add sources:

for a study of north american bureaucracy - 'The Utopia of rules’ by David Graber (my summary here http://mosermichael.github.io/cstuff/all/ramblings/2015/03/2... )

for the study of Russian/Soviet bureaucracy - 'Dead Souls' by Nikolai Gogol http://robobees.seas.harvard.edu/files/gov2126/files/gogol_d...

I have no problem with that though. Banks should try to predict the likelihood of people repaying loans. Giving out loans likely to fail is bad. For the economy, for everyone else taking out loans, and for the people loaning money. It's certainly much more accurate and much less biased than the old system of having a human determine your worth. Same with the IRS using prediction to spend less resources and perform fewer audits.

Putting a number on someone isn't inherently evil. The old system had numbers too, they were just in people's heads instead of in a computer. And as I said, they were much less accurate and much more biased.

China is trying to punish people with different political opinions or behaviors. The american system doesn't care what your political opinions are, just that you don't commit tax fraud and pay back your loans on time.

Oh, come on. We have our problems as a society but this is a little much. I have two specific objections.

* The financial crisis was partially a result of mortgages being given to people whose financial data would have suggested that they had no business borrowing so much money. In that case, the opposite of your point is true. This shows that attention to data is beneficial to a mortgage system because it helps avoid questionable loans.

* My friends don't rate my worth, because I don't have friends.

They weren't given loans because they were incapable of figuring out they were bad loans due to insufficient data they were given bad loans because there was money to be made by giving out bad loans and bundling into mortgage backed securities. They deliberately crashed our economy as part of an effort to inflate their bonuses because they just don't give a fuck about you. me, or America.

You are approximately right, but I don't think that they deliberately crashed the economy. I think that they were hoping for something to turn up and had good reason to think it would - after all, long term capital finance got bailed out, why not mortgage backed securities?

The lesson is that the risks of finance need to be sufficiently crystalised that behaviour is moderated, the problem is that economic growth has come to depend on risky finance.

>>You are approximately right, but I don't think that they deliberately crashed the economy.

Deliberately, no. But they knew it was unsustainable. Many industry insiders, as well as outside observers, likened the system to a house of cards. And when you play with a house of cards, you know damn well that it will come crashing down sooner or later.

That doesn't make any sense. The derivative products were rated AAA. Banks bought these products and put them on their balance sheets. Why would they knowingly throw away money? Three investment banks totally collapsed, and several more barely survived.

>>That doesn't make any sense. The derivative products were rated AAA.

Those ratings were created as a result of significant conflict of interest. You should read John Bogle's treatise on the subject (he's the founder and retired CEO of The Vanguard Group):


"Market participants—now dominated by speculators, not investors—also joined the parade of miscreants, and our professional security analysts failed to do their job of appraising company balance sheets, largely ignoring the huge credit risks assumed by the new breed of bankers and investment bankers. And let’s not forget our credit rating agencies, which happily bestowed AAA ratings on securitized loans in return for enormous fees that were paid in return by the issuers themselves. (It’s called “conflict of interest.”) Yes, there’s plenty of blame to passaround."

This is approximately correct.

I think it is worth noting the subtleties around "they", though. Some people thought it was unsustainable. Others were dumb and didn't. Many didn't care and just wanted to be paid, now.

> The financial crisis was partially a result of mortgages being given to people whose financial data would have suggested that they had no business borrowing so much money. In that case, the opposite of your point is true. This shows that attention to data is beneficial to a mortgage system because it helps avoid questionable loans.

Were credit score non existing in 2008? If so, were they completely ignored?

Also, in comparison, European countries with non existing credit scores didn't have a crisis with subprime mortgages, probably because they paid attention case-by-case to the loan applicants. They surely used data, but it does not mean that this data has to be centralized and behind black box algorithms.

FICO scores existed and were ignored, yes.

Lenders would make money by originating loans and selling them to larger financial institutions, so they had no skin in the game when it came to counterparty risk. Lenders certainly had access to FICO scores, but they had a profit incentive to pitch and approve loans to unsuitable borrowers, and there is a strong argument to be made that they used passable FICO scores to justify bad loans to people with no income and no assets.

https://en.m.wikipedia.org/wiki/Predatory_lending https://en.m.wikipedia.org/wiki/Subprime_lending https://en.m.wikipedia.org/wiki/NINJA_loan

Separately, one of the many differences between Europe and the US leading up to the crisis was that thanks to years of comments from Alan Greenspan, it became culturally acceptable and even encouraged for American borrowers to "tap into their home equity" by taking second mortgages. So, when houses got cheaper, many people who had been paying loans for a while still owed plenty of principal. One might argue that there was negligence on the part of second mortgage lenders here.

> Were credit score non existing in 2008? If so, were they completely ignored?

I think the parent's point may be incorrect in this example. The financial crisis didn't happen because the lenders weren't aware the clients had a high likelihood of default, it was because the lenders didn't care the clients had a high likelihood of default.

The mortgages were packed into packages and sold to other entities so the lenders who initiated the loans passed along the risk. They knew very well that the clients were high risk, but they were incentivized to ignore those risk factors.

Parent here...I agree with you, and if you read my comments again you will see that I never said the lenders were unaware. I said the credit history data were available and would have indicated a lack of suitability for the borrowers.

Ah, I thought you were trying to argue that they were unaware. Thanks for clarifying :)

I think that the lenders were aware and thought that the borrowers were unsuitable but thought that they had discovered a way that would allow high risk loans to be made safely. A lot of really clever Ph.D's had said that securitisation of these loans would work to manage and distribute the risk. I remember going to a dinner party and being lectured by one of these guys and thinking that it made no sense to me, I felt like I always do when I get a new maths problem and I can't figure it out at all and every one else seems to understand it! I was embarrassed by my lack of ability. The said "lecturer" was patently earning several multiples of my salary (I guess that he earned, gross, 10*). He wasn't ostentatious about his wealth, but his watch was nice, not showy, they had come in a new Range Rover, and they lived in the Surrey communter belt. All of this communicated success and authority, his Ph.D came from a big, important university, mine comes from a little and not prestegious one, I struggled to get through my Ph.D - I found it really really hard. I suspect this guy had cruised.

And yet, he was spouting complete rubbish; all the assertions that he made were contrary to straightforward treatments of the data and common sense. I can't even remember the technical justifications he made, I think that I probably didn't make appropriate challenges to get these.

So, I went back to my job as the telco doing pedestrian things with telematary data (n things not working = problem). The lesson of the conversation is that the smart people who were figuring out risk and investment strategies in these places are not smart. Also I was not of the caliber required to smoke that out.

An actionable lesson that I have run with since (ok, I thought this out in a slow way about 18mths after the crisis while chewing over past humiliations and social disasters in which I have participated), is that critical thinking is the most important skill in my portfolio, and I'm not good at it - and almost everyone I meet is worse than me. So I have focused on trying to recruit people who seem capable of thinking round problems, and then have invested time and energy in getting them to be able to understand problems and become domain experts. This is different from the normal strategy of recruiting experts, doesn't deliver results reliably (which is challenging in a business environment - I have missed my scorecard a few times since this experiment began and I am rendered nervous about my career as a result) but has produced a business capability which to me seems qualitatively different from the previous capability that I took over when I got my current role.

Anyway - it makes for more fun at work.

Interesting point. Certainly, the people who put the products together didn't do the math correctly. I think the major problem there is the assumption that homeowner defaults are independent random variables. In such a case, the high credit ratings for derivatives of subprime debt contracts would make total sense. However, if home prices decrease simultaneously and people go underwater on their loans, then suddenly there can be a massive correlation event where even the really highly rated derivative tranches are in danger.

Assumption is a key word. It's hard for people to spot and test assumptions, and it didn't happen in the run up to 2007. Also I think group think is a killer - people go along with the pack, there is often no diversity of opinion or approach. If I ran an investment bank I'd run separate analysis teams, geographically separated and organisationally partitioned. Could save the odd $10billion...

* My friends don't rate my worth, because I don't have friends.

Sorry to hear that.

That's a bit too much hyperbole for me (and the top comment!). Sure, the US has its issues (and, if you're not taking direct action to change it...well). But, live outside the US for a while, and you'll see the difference. For example, live in an ex-soviet country. Experience the doorman to your apartment building taking notes on what you're doing, who you're with, etc. Wait for the doorman and every other 'watcher' to compile this data and report it to your boss. Wait for it to come up in your annual evaluation. I think you'll see the distinction.

Yes, and?

Should we turn credit into a market for lemons? Just ratchet everyone's access to credit down to what's now granted to those with bad credit? Go back to the days of "how respectable do you look walking into the bank?"

Should the IRS spend its finite resources looking for fraud where it's less likely to be?

Should middle-aged people with clean records in Accords pay the same premiums as people 19-year-olds with DUI convictions in WRXes?

Should the state assign friends/spouses by RNG and enforce that you spend equal time with each?

People are different. They have different personalities, different behaviors, different risk profiles. It makes no sense to ignore these things. In the organization examples you've given, at least, it's much better to use data-driven, evidence-based models kicking out scores than to rely on the prejudices and informal gossip networks of gatekeepers - like we did before we were sophisticated enough for records.

Even where scores aren't appropriate, you're going to allocate your resources - like attention - to people you enjoy over people you don't.

China's system is problematic because it's assessing something we think should not be assessed or considered anywhere - loyalty to the Party. But they were likely already doing that anyway, with informal gossip networks among Party officials. Executed correctly, the "social credit score" system should serve that end more accurately, with less corruption and ultimately more egalitarianism. We just don't think the end should be served.

I think the difference is in the secrecy. In the US, at least in principle, the system is not secret - in China secrecy is the rule - the party is not supposed to be judged from outside.

I think the main difference is that you will still get most jobs in the west, even if you have the "wrong" opinions. Like reading Chomsky's collected works in the US. (Or follow Islamist web sites in, well, any country.)

I.e., it is mainly for commercial/security reasons in the Western world, not for political control of the population.

What? No. The vast majority of the stuff the OP listed is secret. You don't know - and have no way of finding out - the various ratings the IRS, the insurance companies, border patrol, the FBI, etc. use to rate you. In fact the only rating you can easily access is your credit rating.

The problem is that in China ratings can have side effects in other ratings. If you are not politically aligned your credit score can go down.

yeah, the one that worries me most is the FBI one because it probably keys off keywords we use on twitter. What a way to chill free speech. As bad as anything that China might do

Whats the alternative?

Same goes for propaganda and media control.

I just finished watching the Nosedive (Season 3, episode 1) from Black Mirror on Netflix, it shows a similar system. Unlike other Black Mirror episodes its quite well balanced. I would highly highly recommend watching it.

I watched the first 3 episodes of S3 last night, and as soon as I saw the headline for this story, I thought of Nosedive.

Watching Nosedive, I thought how close to it we were already, it's just that we're measuring people in terms of Followers or Likes. This path started I think when we prioritised Google Analytics numbers over the positive effects you create within your audience: clicks, traffic, unique visitors all seem to count more in 2016 than loyalty and shared values that dominated publishing for the prior 100 years or more.

In the age of "personal brand" it makes sense to me that we're going to end up with "personal metrics" and we're just seeing the tip of the iceberg.

At the end of Nosedive I decided if that was the future of my society, I'd stay being me and suffer the consequences. I'd rather be happy in my own skin that be popular, and if I lived in a society that turned popularity into a measure of personal worth, I'd quite happy to be a dissident within it.

I watched it last night and was struck by how much impact had been lost by Netflix transition. OK they were clearly going for humour, but even with bigger production, and good acting the longer running time diluted it so much. We didn't need the star rating explained every 2 mins for the first 20.

Though I think it was that one that's based on a Charlie Brooker story, rather than written by.

I actually paused after to decide if I was going to bother with the rest at all. 2nd episode (as far as I've got) was much better.

I just finished the third and have liked all three but agree with you completely. The original runs packed far more punch than what I've seen so far.

Strike that, the fourth episode is amazing and as good as any made previously.

My first impression and suspicion is that it has been Americanised. British humour tends to be more grating and grotesque than what's considered acceptable in the US, and the outlook is bleaker. So far I've watched the first three episodes. The first even has some sort of happy ending. The second episode looks just like a bleak prequel to Inception, rather predictable. The third is maybe the most faithful to the original spirit. There seems to to be a slow transition from American to British in the three episodes, so curious to watch the others.

It started like that, but I quite liked San Junipero and Hated in the Nation. The latter was on a similar par to White Christmas, in my opinion.

It's Always Sunny In Philadelphia used this concept as a plotline, and managed to treat it with more subtlety. Granted, it's a comedy, so the tone was wildly different.

Which episode?

The Gang Group Dates

My favorite aspect of this episode is that the main character's drastic decline in quality and ease of life that coincided with her declining rating are the exact same experiences someone with poor credit or a rap sheet would experience right this second. The rating systems, lists and records we use as a metric for access and worth are draconian as it is. Crowd sourcing it just makes it glaringly obvious with a tight feedback loop.

Just the title of this post reminds me of the anime Psycho-Pass


First thing that comes to mind for me whenever Sesame Credit is brought up.

I just finished watching it myself, after having seen it mentioned in your comment. I found it profoundly heavy-handed, and, unlike pretty much every other episode of the series I've seen so far, I didn't enjoy it at all.

I really disliked this season. It's too sanitized, not anywhere as visceral or cringe-inducing as the previous ones. Hell the first episode actually had a happy ending instead of a deeply depressing one; the endings in Black Mirror always had that appeal in that they always uneeringly hammered in that there's no escaping, there are no happy endings and it can only get worse.

It just felt too "plasticky", that conclusions about the worlds displayed were being forced on me, instead of giving some room for thought.

It was my favorite season. I probably won't watch Episodes 1 or 6 again, but I thought the middle ones were consistently great. The one with "roaches" I seemed like it had excellent warning value as we enter the world of augmented reality.

Not trying to start an argument over matters of taste, just interesting to see how opinions differ! It does seem like in season 3 -- aside from the episode everyone is talking about in relevant to this thread -- they run with more of the "sci fi premise" approach, as opposed to making you feel like this is a just around the corner, as some of the first seasons did.

Came here to mention exactly this. The 1st world isn't far off something similar (instagram/twitter followers represents power and popularity)

Same here, they are doing some very interesting design fiction stuff on this show.... Not always recommend to watch before bed though :)

watched the first three episodes - still can't sleep (its 4am).

After finishing the season today, they've really stepped up their game, it seems, from the previous seasons.

Exactly. I give you 5

2. Wasn't a meaningful encounter.

:D Ha!

My concern is that China will use the scoring to make moral judgments about citizens. Yes, we have credit scores and DMV scores and some of us wind up on white lists (PreCheck) or black lists (No-Fly) for air travel.

But determining business risk is different from determining whether someone is a moral or immoral person, and then using that subjectivity to control the population. It's value to the regime is in that subjectivity: If you threaten my political aspirations, I'll score you as immoral, report your transgressions on the nightly news and toss you in prison to be reprogrammed.

The US separates church and state to prevent this.

The problem with this happening in China is that, given the rampant corruption, authoritarianism, lack and transparency and lack of rule of law in China, the system is pretty much guaranteed to be expressed in the worst possible way: used for population control and political repression, and gamed by those in power for their own benefit.

Something really bad is happening in China. There's been talk for a few years about abolishing the hukou system of housing registration, which would appear to be a step towards liberalization, but at the same time the government seems to be running full tilt in the other direction, developing new systems for control. I just left China after fifteen years, and when I moved out of my apartment in central Beijing, an Indian guy was considering moving in after me. The local police station told my landlord, "no Indians". Later they told her, "no one without a Beijing hukou can live there", which is pretty staggering if true, as the majority of Beijing residents don't have Beijing hukou.

I don't know what's going on, but levels of paranoia in the higher reaches of power are spiking, big time.

> The problem with this happening in China is that, given the rampant corruption, authoritarianism, lack and transparency and lack of rule of law in China, the system is pretty much guaranteed to be expressed in the worst possible way: used for population control and political repression, and gamed by those in power for their own benefit.

Well said. This can be said about pretty much any "system" being introduced in China.

The only "morality" the Chinese government cares about is whatever leads to the furtherance of those in power

You're worrying too much. A big differerence in China is that people will take this kind of things much less seriously. Even the impact of credit scores are close to zero.

What does church have to do with this? We can throw people in jail for "corruption"

In the U.S., the courts can't decide whether something is moral or immoral. If we throw people in jail for corruption, it won't be because they were found guilty of having a bad moral character. It will be because of an illegal conflict of interest or anticompetitive practices or something like that. That's what a separation of church and state means in this context.

The US court sure comes close to deciding moral issues, if not actually. Abortion: some people say killing babies is wrong and should be illegal; other people say a woman has the right to kill* her baby if it hasn't been born yet. Homosexual marriage: some people say it is wrong and should be illegal; others say it is right and should be legal. So if I think homosexual marriage is wrong, the court requires that the society I live in treat it as okay, which leaves many people feeling like the court is legislating morality. (Of course, the opposite is true, too) Furthermore, the court pretty much has to rule on things like this. So it's a pretty fine line.

* I freely admit to being biased on this issue, but the fetus has a heart-beat at 3 weeks; it's hard to say it's not living being. And if it's living, aborting/terminating it is technically killing it.

No, this isn't true. This isn't how the US courts work. The US courts decide what is consistent with the law. Let's look at the two issues you're discussing, abortion and gay marriage. These were both decided by Supreme Court cases, Roe v. Wade and Obergefell v. Hodges, respectively.

In Roe, the court found that outlawing abortion would constitute a violation of rights already established in the Constitution, specifically the due process clause of the 14th amendment. This means that the same legal rationale for your right to abort is also the legal rationale that prevents a state from, say, passing a bill stating that "prewett is now a criminal because we say so."

In the case of Obergefell, the court decided that denying recognition for same-sex marriages from other states was a violation of equal protection, because opposite-sex marriages from other states were recognized. This is the same right that, say, allows you to marry someone of another race and then go to another state and have that marriage continue to be valid.

So no, the court is not deciding, morally, what is right and wrong. If you think that, that's probably the source of your confusion. The court is deciding that the implication of the constitution and laws as written demand this conclusion in order to be logically consistent.

You might disagree with the logical (or moral) basis of their decision. Sure.

You might say they're legislating morality. No.

I think I see what you're saying, and I agree that laws and morality are similar. That said, I do think they can be meaningfully distinguished. Edit: I understand you make this distinction as well.

The duty of the judiciary is to interpret and apply the law. The creation of the law is the job of the legislative branch. It seems to me that while people may agree or object to laws on moral grounds, the judicial branch is focussed on the application of the law itself, not the moral aspects of it. As the legislature creates the law, one could argue that they're more directly responsible for encoding morality into the law. Though some might think in some cases the judicial branch abuses it's interpretive duty by "legislating from the bench".

What do you think? Edit: I suspect we're actually quite close to agreeing.

I think we pretty much agree. Rights and morality are pretty closely related, so when the court rules on rights, it's pretty close to morality.

I think what upsets people about "legislating from the bench" is things like at least 25 states pass laws or constitutional amendments banning some form of homosexual marriage [1], which clearly reflects a substantial will-of-the-people, and the Supreme Court overrules the expressed will of the people and says it's unconstitutional. So now these people, have to live in a society that legally tolerates something they think is wrong and/or harmful to society long-term. Regardless of your view of whether the Court was "right," you can certainly see why people would be upset with this. No matter what the Court rules, someone is going to be unhappy, but given that homosexuals are 3% of the population and are driving the other 97% (which had expressed their desires through the democratic process), you can see the problem.

(I think this is why conservative Christians are voting for Trump in large numbers, despite the fact that many of them think he is not a good candidate. They are voting for a Supreme Court nominator, not a government leader. As a conservative Christian myself, I think this is foolish, but there it is.)

[1] https://en.m.wikipedia.org/wiki/List_of_U.S._state_constitut...

A lot has happened in the past decade, and it's not 3% driving the 97%, not that it ever was (http://www.pewforum.org/2016/05/12/changing-attitudes-on-gay...).

But either way, the percentage of people who believe certain morals does not matter. No matter what our individual beliefs are, we bind ourselves to the law of the land. That's what it means to have a constitution. Even if 100% of Americans wanted to take away your freedom of speech or your freedom to bear arms, we could not do it legally.

The 3% is the percentage of people who are homosexual, not the people that support them, but I see your point.

In this discussion I'm not taking sides, just noting that the court needs to decide quasi-moral questions. Our interpretation of the Constitution depends on our current beliefs (and might even lead to an interpretation that the original founders would have rejected). The constitution clearly says that we have a right of free speech. However, as far as I am aware, it does not state that we have a) a right to marry, b) a right to do homosexual acts, or c) a right for homosexuals to marry. In fact, I believe B and C were illegal for large periods of time (anti-sodomy laws). I suggest that the Founders probably would not have seen homosexual marriage as a right. Due to changing morals, however, the Supreme Court now views it as a right. Were the laws constitutional before but not now? Were they always unconstitutional, but nobody challenged them? As a thought experiment, if societal morals changes to believe that homosexuality is actively harmful to society, would anti-sodomy laws be constitutional? The process of interpretation of rights necessarily involves our current beliefs.

Can it survive on its own outside the mother's body? Or implanted into another mothers body? If not how alive is it? But more importantly how can you enforce upon a woman the sacrifice to her body and self to carry this fetus to term? Does she not have autonomy and control of her own body?

This is what the DMV does with the point system. The FBI and CIA I think also collect statistics and profile relevant individuals. The difference I guess is that most of these systems are in separate silos. The Chinese government is taking the next logical step and linking it all together. Reminds me of a quote

Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.

DMV tracking whether you are driving drunk is not the same as the Chinese govt. tracking whether you praise the govt. on Facebook.

> The FBI and CIA I think also collect statistics and profile relevant individuals.

Citation needed (especially that they do it for everyone).

I didn't say everyone but going by all the leaked documents on various surveilance programs what other citations do you need?

> > relevant individuals

> Citation needed (especially that they do it for everyone).

1. We know that they can. If you were designing a mass survaillance system, you would store all incoming information into unique bins now, then later select on the bins of interest.

2. Three years ago I asked an ex-govt employee who would know. When I got to that exact question, he got visibly upset and abruptly ended the conversation.

The US does track what you say on Facebook to see how much of a terrorist or criminal you might be. Just because you think "praise the government" isn't worthy of reward, doesn't mean it isn't. Afterall, China doesn't have a democracy, so if people are too dissatisfied with the government, instead of voting it out, they might have a revolution, which would be disastrous. In that environment, praising the government can be helpful to keep people safe from violence.

I agree it makes sense for the Chinese govt. just as it made sense, for example, for a slave trader to put down slaves that didn't toe his line and reward those that obeyed him fully.

You know what might make more sense? The Chinese govt. (or any other similar aspiring govt.) should spend billions on BCI research and implant chips in everyone so that any dis-harmonious thoughts can be culled right at the source.

China lacks almost any societal pressure valves at all without online dissent and armed revolution is all that's left if people become unsatisfied. "Praising the government" makes people less safe, since honest real public sentiment is well obscured.

Points aren't really big data, they are based on rare, high signal events rather than small statistical differences across many observations.

Sure but on the spectrum of tracking data where exactly are you going to draw the line? I'm not saying I agree with what they are doing but clearly these kinds of systems already exist. This kind of reward and punishment system exists pretty much in any large enough organization. What is an internal review and promotion system if not some kind of systematic tracking of data to reward "good" behavior?

If the Chinese government is viewed as one giant organization then again this is just a logical extension of such systems.

They are just taking a cue from credit rating agencies.

I would not be okay with my company rating me based on ML computed metrics of my performance based on everything that I do in (and out?) of the office

How is a credit rating different from citizenship rating? Your buying patterns reveal an awful lot. Why is one ok and the other isn't?

People's lives should not be affected by how loyal they are to the state (up until the point of actual violence).

People's access to credit should be affected by how they behaved with previous loans.

Credit ratings are about how risky it is to lend you money, not where you are allowed to live, drive, and work.

They often work out to the same thing. I've been denied work and housing due to credit rating problems.

Is that really true? Access to credit determines a bunch of things and I'd say it's not just limited to where you live, drive, and work. It can sometimes even determine whether you're able to pay for healthcare or not.

As I understand it, the reason the world have become more democratic and liberal for generations was because freedom made a society work better. The open societies out competed the closed ones.

If this model really works for China and other non democracies, so they can keep a liberal economy and still control people hard politically, then:

1984, here we come. :-(

It really does seem like, with so much distraction and so little necessity, we're nearing the end of the democratic stage of the cycle, where people use freedom chaotically.

No, the world became more US like for generations because the US was a relatively homogenous ethnic population (after genociding the competing ethnic groups) that controls a quarter of the planet's natural resources and has oceans for borders, so it rose to economic and military dominance easily.

Trackers and social media companies try to build similar scoring systems here.

If banks or insurance companies can figure out how credit score changes of Facebook friends predicts you financial status, better abandon them if they screw things up.


Project Cybersyn in 1970's Chile, http://www.newyorker.com/magazine/2014/10/13/planning-machin...

"Beer was a leading theorist of cybernetics—a discipline born of midcentury efforts to understand the role of communication in controlling social, biological, and technical systems. Chile’s government had a lot to control: Allende, who took office in November of 1970, had swiftly nationalized the country’s key industries, and he promised “worker participation” in the planning process. Beer’s mission was to deliver a hypermodern information system that would make this possible, and so bring socialism into the computer age."

Cybersyn was about giving the government what we would call a 'dashboard' of the planned economy (with Star Trek chairs), not social control.

How incredibly dystopian of you, China. This is also pretty funny considering it is exactly the premise the first episode of the new Black Mirror season.

Well, I recall news from at least 1-2 years ago about these Chinese plans.

This reminds me of the "Whuffie" social currency from Cory Doctorow's science fiction novel Down and Out in the Magic Kingdom. A person's current Whuffie is instantly viewable to anyone, as everybody has a brain implant giving them an interface with the Net.


Which, incidentally, Cory Doctorow considers to be a fundamentally terrible idea: http://www.locusmag.com/Perspectives/2016/03/cory-doctorow-w...

my first thoughts exactly. It seems more and more that we're heading towards this type of system.

Reddit and YC also expose each action to voting and judging, and sum up users into karma scores. We already live in such a rating based moral system. If the reddit karma would decide a person's worth and social rights, that would be pretty dangerous. People would self censor, be less creative, and in general only act in a way that protects the score. It would turn into a kind of self policing.

True, but these are privately owned communities and if you disagree with their standards you can go join some other community or start your own. A Chinese citizen can't exactly go start their own competitor to the Communist Party.

And yet... credit scores are maintained by private companies, and are inescapable in the US, as are their effects on your opportunities in life.

So "It's a company, not a government" is not a useful argument to make for this kind of situation.

The credit rating industry in the US is very heavily regulated. Setting aside whether the regulation of that industry is good or not, there is really only one US credit scoring system and it's effectively controlled by the government, so this is not an example of multiple, privately controlled, competing systems at work.

The Chinese "social score" system would almost certainly be developed and administered by private companies (prototypes and precursors have been). The government would just set the rules. So in fact it would be similar to the US credit system in this way, one system, dictated by the government, private entities just handle its operation.

And to combat the flaws in the private system the government passed laws giving you the right to see the data they have and challenge that data. Are you arguing that statistical analysis should be outlawed? Do you think everyone should have to pay more to borrow money?

Someone mistook various dystopian science fiction books for howto guides.

Well, this is bad because it's the Chinese government.

But companies like Airbnb have to use proxy metrics like number of Facebook friends. If I could be proven to be the trustworthy person I am, then a host wouldn't need to look at all that. If I'm known to have destroyed someone's home, then someone else on Turo knows better than to rent to me.

In practice, it's likely to turn into an eBay style "A++++ best person ever. Let me on the bus brilliantly! 5/5" but the idea isn't terrible on its own.

>Well, this is bad because it's the Chinese government.

Follow up question: would this be good because it was by the US government?

No! Too much power and information in one place is always bad

I'm not in favour of this being administered by governments but I'd like it to be based on a government-issued identification number. Like if we split the identification and authentication roles of the SSN into two separate numbers. Then I'd like it to be based off the identifier portion.

Wait Airbnb is gonna discriminate against me because I boycott Facebook?

Not Airbnb. Your potential hosts. Airbnb allows you to provide multiple means of proving you're reasonable. Linking your FB profile is one way. Airbnb will then show the number of friends you have.

This may have changed since when I signed up years ago.

It's just like your GitHub is a reputation aid when applying for a job. It's not required but what you have on there can help.

EDIT: In case it isn't clear, I'm not claiming here that Airbnb is forming some sort of transcontinental people database.

"number of friends"? That's stupidly gameable

I.e. nothing near what China is doing.

You're insane. This is about having full control over the population.

Imagine a world where saying something bad about a company like Airbnb means you won't get a loan from the bank because the owner is a communist party member and your worthiness score is now reduced to an unacceptable level.

This is a bad idea not because it's the Chinese government. The Chinese government is bad and this is just one more terrible idea that shows us just how far they are willing to go to control the population in perpetuity.

Hey, in China being a member of the Communist Party helps you get a loan.

Did you even read my comment? In my hypothetical situation, Airbnb was owned by a communist party member. A person said something negative about the company and had his/her overall score reduced.

This is clearly a very bad direction for China in itself. Most people reading this instinctively want a "but", and this is one that I can provide that applies to China and Russia (though Russia is really much freer than China):

Western critiques of China and Russia are directed at the regime, but if the regime collapses, there is no on to criticize, but people still suffer. That is, the Western critique is not fully utilitarian: it criticizes the regime for taking measures to ensure stability, without considering the consequences of the regime losing control.

For reference, about 70 million Chinese died after the collapse of the Qing dynasty (caused in part by Western intervention, by the Opium wars), and Russian life expectancy reduced significantly during the power vacuum between the fall of communism and the rise of Putin.

This gives some background on why both countries support their governments: they want stability and strong central government because they know the alternative.

If Westerners want to adopt a moral and convincing approach to human rights in China and Russia, they need to explain how these things can exist without causing a collapse of law and order that would be worse for the ordinary citizen. This would require reigning in the CIA so that countries could loosen the reigns without fear of a color revolution or Arab spring.

With Trump and Hillary, the community on Chinese websites are increasingly disenchanted with democracy. One high-vote answer in Zhihu, or "the Chinese Quora", claims that this election cycle has been a "great educational experience for Chinese Nationalism and Marxism". You'd be surprised how popular Trump is in China, and how much they support their Government.

If Chinese love Trump, why are they disenchanted with democracy? Trump is doing great in democracy, in the #1 or #2 spot. And I'd assume they'd like Trump less if they were more knowledgeable of English and USA news.

By democracy, I think he meant hillary's brand of "democracy", which is going to win in November.

Stability and strong central government are on opposite sides.

"Imagine a world where an X government monitors everything you do, amasses huge amounts of data on almost every interaction you make, and awards you a single score that measures how “trustworthy” you are."

What country is he talking about?

This is wonderful. A fully gamified life. It is so dystopian...

America has the credit rating system. With much hand wringing and energy devoted to increasing or altering the score and so on (companies advertising a way to increase it for you, landlords claiming they'll ruin your credit rating if you ask for a deposit refund ...).

Now imagine if the system was even more corrupt and you could pay someone under the table to alter your score directly. Now imagine that is not just a number for your credit worthiness, but a number for your, well general-life-worthiness. There is no way that is not getting exploited, hacked into or gamed. Maybe that is the ultimate goal, it lets those who control this system, control the population?

Imagine an implied threat of getting -150 points for joining a protest. Or maybe your relative is going to court, you imply to the judge he might get a +50 if he lets your nephew walk. Possibilities are endless, as they say.

Isn't this the case already? The judge will let your nephew walk if you are rich enough. You don't need to be in China for that to be the case. What is wealth if not another point system?

It is just much more efficient if you can just twiddle a number in a database instead of say having to gift them a house. Or instead of capturing and torturing someone, just lower their "karma" so they can never get a job first. It is about efficiency.

We already do this. Everyone has a credit score. Everyone gets a background check before renting or getting a job. Let's not pretend we don't have a rating for everyone yet...

I think the moral highground to judge surveillance and human rights issues has been lost. Pre snowden this kind of thing would cause mass hysteria, massive moral grandstanding and call for sanctions in western populations and media.

Now chastened its more of legalese and tiny triumps based on hope rather than scrutiny.

Just as well the sordid culture of using human rights to score moral points and signal cultural superiority is done and the remaining vestiges of apologism and denial should think seriously about self examination and scrutiny lest the ground shifts even further under their feet.

Coincidently, an article[1] just came down my WeChat pipeline about the population clean-up of Beijing. It's a great, classic propaganda article, full of pictures and statistics about how crowded Beijing is, and lots of "wouldn't it be nice if there were fewer people?" and "look how hard your government has had to work, don't you feel sorry for us?".

Basically they're driving out those who don't have Beijing housing registration. The propaganda is addressed to those who will enjoy the relief after the undesirables are gone. No one speaks to the undesirables.

The one-child policy may be over, but the Chinese government's manipulation of the lives of its citizens is alive and well.

[1]: https://mp.weixin.qq.com/s?__biz=MzA3Mzc3MTc0Mg==&mid=265293...

Edit: Here's another link (sorry, also in Chinese) from June, laying out the bones of the policy to drive people out of the capital. Again, posed as a great policy with no downsides. http://www.china.com.cn/guoqing/2016-06/17/content_38687237....

Relevant extra credits episode: https://youtu.be/lHcTKWiZ8sI

I'm halfway through "Weapons of Math Destruction", and this sort of thing is really, really scary. If the amount of damage done in the US already with big data models, with a much narrower focus, for smaller consequences, and ostensibly "good" purposes is any indication, this will be incredibly harmful.

I feel sad about the Chinese people and people living under communist/Marxist regimes.

Communism/Marxism is one of the worst ideologies out there (along with intolerant religions). The western democracy is the best model the humans have invented yet. This is not to say we must not strive to improve it. But Communism/Marxism is not at all an improvement, it's an outright societal cancer. But I am seeing is the increase in criticism of US (and western democracies) by leftists and liberals. That in itself would not have been a big deal but these leftists praise Communism/Marxism.

This is a wake-up call to the people of united states who are listening too much to the unjust and undue criticism of the US govt for surveillance. Post-Snowden some leftists are taking undue advantage of the atmosphere to spread their fear mongering anti-democracy propaganda amongst US citizens.

China is not a communist state these days nor I'm not aware of mainstream Western liberals praising communism. (sources on the latter?)

>>China is not a communist state these days

That's your opinion. FYI, China has the dreaded communist politburo running the show.

> This is a wake-up call to the people of united states who are listening too much to the unjust and undue criticism of the US govt for surveillance. Post-Snowden some leftists are taking undue advantage of the atmosphere to spread their fear mongering anti-democracy propaganda amongst US citizens.

I don't know why you would equate opposition of mass surveillance with "anti-democracy propaganda". A major point of opposing mass surveillance is that it's anti-democratic for the government to be able to dig up dirt on anybody with the press of a button.

You are total wrong. In fact,today China is not communist/Marxist at all.China is more Capitalism than USA. The word communist/Marxist is only mentioned when holding big meetings.

China is definitely not communist, but it is definitely less capitalist than the US. Fundamentally the "Communist" Party is about control, which is directly opposed to a free market (i.e., capitalism). For example, the Chinese government wants a stock market. But, they want this market to always go up. (down = people lose money = risk of social unrest) So this year they tried stopping the market if it fell too far, suspending trading of certain companies, telling fund and/or brokerages that they could not sell certain companies. NOT capitalism. Also, the economy is dominated by large state-owned-enterprises (SOEs) which receive strong direction from the government. The government tells factories around Beijing to close whenever a government meeting is in town, so the air will be clean. Note how none of those things happen in the US.

>>China is more Capitalism than USA.

China is not at all capitalist as there is no free market, and you may say it's not communist/marxist if it pleases you but it's a communist party controlled tyranny. *

If anything, China is much, much more tyrannical and dictatorial than USA. Tyranny and brutal dictatorships is only what communism/Marxism have produced in large amounts. The communism/Marxism have produced very little good and very large evil wherever they got foothold.

* A Chinese citizen cannot even move within China without getting permits from the communist politburo approved officials, forget about him/her being able to launch a business freely.

Edit: added * details

Communism/Marxism =/= tyranny and brutal dictatorships. They aren't mutually exclusive, but one doesn't imply the other. There was Marxism and tyranny in the Soviet Union, modern China only has tyranny.

The modern Chinese regime has nothing even remotely close to neither communism or Marxism except the imagery.

Frankly, I think it is the far Right that are closer to anti-democracy. Witness: Trump claiming he won't accept the results of the election, that's about as undemocratic as you can get.

Where did Trump claim that he won't accept the results of the election? In the debates, he refrained from saying one way or another.[0] There's a big difference between what he said and what you're saying he said.

[0]: Trump: What I’m saying is that I will tell you at the time. I'll keep you in suspense, okay?

Read more: http://www.politico.com/story/2016/10/full-transcript-third-...

I think it is very undemocratic to even suggest that that you might not accept the results. In fact, it is even slimier, because it provides maximal disruption in trust of the democratic system while allowing him to claim exactly what you are saying.

This is kind of interesting, in a "history of communism" type of way.

There’s been a sense that communism as a goal in China has been in zombie mode for a while with the system continuing temporarily until an economic crisis or somesuch brings it to an end.

This reminds me of the various “rationalisation” initiatives undertaken by state communism(s) in their early days which formed such a big part of what we think of as communism. These were (attempts at) nitty-gritty policies aimed at achieving the goals of big talking communist slogans and utopianism.

It seems revivalist.

In some ways, this ideas kind of makes “sense’. Digitisation, social media and other features of modern societies lend better (at least in theory, while wearing our 99 year old state-communism hats) to this sort of societal engineering than their 20th century equivalents.

Zombie mode communism has been already tried right in the beginning by New Economic Policy in Soviet Union between 1922 and 1928 https://en.wikipedia.org/wiki/New_Economic_Policy

The end was one of the bloodiest crimes against humanity and one of the most widespread genocides of the previous century.

"Imagine a world where an authoritarian government monitors everything you do, amasses huge amounts of data on almost every interaction you make, and awards you a single score that measures how “trustworthy” you are."

Oh, like the US.

I despise this aspect of our field. On the one hand we're ushering in a new era of learning and knowledge. On the other we're creating tools of unprecedented power.

They're searching for enemies of their state and they will probably find them. Let's not ignore the fact that heroes and patriot's often start out as an "enemy".

Sarah Winchester believed she was haunted by the men, women, and children killed by her husband's invention. What will keep us up at night in the future?

The North Koreans already do this, and they did it with bureaucrats, not "big data" https://en.wikipedia.org/wiki/Songbun

http://www.hrnk.org/uploads/pdfs/HRNK_Songbun_Web.pdf (PDF with more details)

I've seen reports of this for at least 6mo to 1yr. Is this vaporware by China or do they have anything more than a press release at this point?

This is something beyond the sesame credit score talked about before.

It is already in place. If you owe someone some money and you don't want to pay. He can sue you in court, and what the court does is that before you pay back whatever you owe, you cannot buy flight tickets, cannot take high-speech train, cannot stay in good hotels (basically any where you need to show your ID). agencies cannot book you holidays abroad and so on.

> cannot take high-speech train

This is one of the best Freudian slips / unintended puns that I have ever read. Damn right he's not taking the high-speech train anywhere in China by standing on a soapbox and complaining about Big Brother and disrupting the harmony of society.

Playing devil's advocate here. Is the outrage towards this kind of system essentially a criticism of the technology? Specifically, can we guarantee that the system would be secure and can we guarantee that the system is accurate (objective) and not subject to human manipulation?

If we hypothetically assume that the system is hacker-proof enough and accurate enough for all practical purposes, is there still an underlying moral reason not to implement this system? I instinctively feel that it is wrong, but I just don't know what it is.

I think we all unconsciously rate people in our social interactions. Maybe to explicitly quantitate that rating solidifies it and makes social mobility harder? Our fuziness in our unconscious rating of people gives leeway for others chances to redeem themselves or not? Whereas in a system like this, people are more likely to be stuck where they are?

Propaganda Games: Sesame Credit


All western reporting about Sesame Credit has focused on privacy. Has there been any objective analysis on projected economic impact of the system?

Black Mirror S03E01 minus the government. That show is incredible.

Next up: Attach the score to "basic income".

Next up: Attach it to "basic income".

We have Facebook for us to do this.

Black Mirror - Season 3

Lets all pretend that the US won't be doing this in 10 years.

Facebook/Google are already working with credit agencies and law enforcement.

We make China out to be the big bad guy, but all they do is follow our example.

Credit agencies?

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact