
Chelsea Manning: The Dystopia We Signed Up For - frandroid
https://www.nytimes.com/2017/09/13/opinion/chelsea-manning-big-data-dystopia.html
======
ncr100
One point seems to be about trust. She's noting how society depends upon
digital, not analog, trust.

She thinks of that as dystopian because of the foreignness and inconvenience.

We, as technologists, can improve upon this by talking empathetically more
with our users about how they interact with our technology, to help steer the
product towards incorporating more human-focused designs. This could help
avoid this Stranger in a Strange Land phenomenon becoming more prevalent, and
avoid the negative psychological behaviors which disconnected individuals
typically exhibit.

~~~
sp332
But what if there isn't any big incentive to do that? It's profitable to take
away people's privacy and there's not that much money in letting them keep it.

~~~
barce
Their really isn't any big incentive to do that. Software is still eating the
world.
[https://www.wsj.com/articles/SB10001424053111903480904576512...](https://www.wsj.com/articles/SB10001424053111903480904576512250915629460)

~~~
sp332
I'm kinda OK with software eating the world. That doesn't automatically mean
concentrating power.

------
meri_dian
What she describes doesn't sound like a dystopia at all. Some systems that she
interacted with weren't able to handle her name change. Ok... having some
trouble handling edge cases is not dystopian, that's just poorly designed
software.

It's not like she was permanantly shut out of society. She was temporarily
inconvenienced.

~~~
jimktrains2
Name changes aren't an edge case. What's dystopian to me is the unwillingness
of people to actually work out and solve issues for their clients, customers,
and citizens. "Computer Says No" is used to avoid having to work and avoid
having to escalate.

~~~
2muchcoffeeman
It's not just a name change. She was in prison for several years.

When she got out, she tries to fix up her accounts.

If someone walks up to a bank years after no activity and wants a name change
(and maybe a gender change) on an account should the bank just do it? Probably
not without 3rd party verification that the customer has to initiate by
filling out some forms and providing evidence.

I'm just speculating but it could have been reasonable for the bank to have
declined the request up front.

~~~
jimktrains2
Sure, but the response shouldn't be "sorry", it should be "here's our process.
Thank you for the letter from a judge changing your name, we'll confirm that
with the courthouse and should be good to go."

~~~
2muchcoffeeman
I am not trying to excuse rudeness. Only trying to point out that this was an
extraordinary case.

I bet if it was a woman that got married and changed her last name, it would
be super simple.

------
iamleppert
I experienced this recently in the scariest of circumstances -- a hospital.

I have hemophilia, a bleeding disorder. Normally, its a nuisance that just
requires I take medication on semi-daily basis. If I don't take the
medication, I'll usually be fine, except if I have an active bleed, which will
just continue to bleed, causing damage to my body, and can be life threatening
depending on the location of the bleed.

I was recently traveling back home and I ran out of my medication. It's a
specialty medication that you can't find in pharmacies. It normally has to be
shipped to me. I had ordered some but it wouldn't arrive for another day.

I was having an active bleed in my torso, of all places, which is an odd place
and distressing because its close to so many important body systems (the
spine, major arteries). The thing about my condition is that normally bleeds
are innocuous, but you never know. I always be on the safe side, because when
I haven't its often meant months-long recovery.

Anyway, I figured no problem, I'll just go to the hospital ER and get some
medication, which is what I'm supposed to do and I've done before -- but not
in a long time. Most all major metro hospitals carry my medication.

I went to the hospital and told them about my condition and what I needed.
They wanted me to take a CT scan, which I declined (I already knew what I was
dealing with and just needed my medication). Then, they wanted to test me for
hemophilia (I've had the condition since birth), a very expensive and lengthy
test that takes hours.

After finally speaking with a doctor, he declined to give me my medication
until I both consented to the CT scan and the test for hemophilia (despite me
having official medical identification that explains my condition). I didn't
want the CT scan because its ungodly expensive and exposes your body to a lot
of radiation (100x an x-ray). You could clearly tell it was a bleed due to
swelling, bruising, heat, and my prior medical history. And I've lived with
the condition for awhile so I am very attuned to these problems.

I experienced what I can only describe as a similar type of hostility as
someone coming in seeking drugs to get high (my medication neither makes you
high nor is dangerous, except when I don't get it).

Curious as to why I was being treated this way, I pressed further. The doctor
came back with a print-out from some medical "AI" software. It had not only
determined what the proper course of action was to be (the tests before my
medication), I was also informed that if the staff deviated from the "process"
at all, they would risk being fired.

There you have it. All decision-making ability had been revoked from even the
doctors. They were prepared to let me bleed for hours, while waiting on a test
to confirm a condition I've had since birth rather than risk deviating from
some computer-generated "AI" policy. Even more sinister, since it was a
screenshot (done with print screen) of the windows-based software, I spotted
something else: something called a "risk score". It was a gauge-like
visualization that had rated me in the "riskier" category. No doubt this was
also the case with patients who are drug seeking, and I hypothesize this was
the reason for the apathy and apprehension I detected.

It kind of reminds me of the Critical Care episode of Star Trek: Voyager,
where patients have been reduced to a "TC" \-- treatment coefficient that
determines who gets good care and who doesn't and is based on the person's
accomplishments and value to society. Honestly, we're not far off from that.

What would happen if I came into the ER with a serious problem that required
immediate treatment? Would they defer to this software and watch me die while
waiting on tests? I know my condition is rare, I'm used to educating doctors
about it, but the fact that the doctors seemed unwilling to even compromise
and do the right thing was alarming to me, and also sad. Its like they have
become customer service representatives of a large corporation and they know
it, and so they have divorced themselves of any care or humanity at all.

I decided to leave and take my chances on waiting the day, rather than sit
there for hours. At that point, I was in extreme pain and I honestly didn't
know if the bleed would resolve itself or not. Worse case, I could always come
back to the hospital. I guess.

The bleed ended up resolving itself, and I eventually got my medication and
was able to treat myself. Ironically, a nurse called later the next day to
"check up on me". My tests had finally come back over a day later and
confirmed my condition, and also that my factor level was low. After I told
her the story I've just shared with you, all she could offer was to "make a
note in your chart". That no-one will be looking at, and likely not influence
the medical software at all. I looked on the hospital web site for contact
information to try and lodge a complaint -- there was none. I also tried
calling the hospital's 800 number, which was entirely automated and offered no
ability to talk to anyone. In case anyone is wondering, the name of the
hospital is Riverside in Columbus, Ohio (part of Ohio Health). It's a huge
hospital.

Anyway, this is my story about AI. Something to think about because as
engineers we are the ones making these kinds of software and have an ethical
and moral responsibility. We need to be aware of how people are going to use
technology and actively work to prevent abuses, and hopefully build this into
the software itself.

What kind of science fiction novel are we living in?

~~~
ams6110
Sorry for your experience but a great story about why we should never hand
over all authority to an AI.

I wonder why (a) they couldn't just call your regular MD to verify your info,
or (b) your MD could not phone in a prescription to the hospital's pharmacy?

~~~
iamleppert
They verified my condition with my doctor and medic alert, which has a copy of
my prescription.

The problem is my medication needs to be reconstituted (fancy word for
"mixed") and is given via an IV (a shot).

I can and do both of these things, but with these kinds of medications they
simply can't just give them to you -- they have to be "administered". That's a
topic for another post, but it explains why they just can't give me the box of
medication and supplies. Probably some kind of liability issue.

------
turc1656
Industrial Society and Its Future.

The miniseries on the Unabomber that just ended this week could not be more
timely, even after 20 years.

------
caiob
So, I am reading an article written by Chelsea Manning about trust and
confidentiality. Am I reading this right? Is this where we are at now!?

~~~
shallot_router
About trust and confidentiality... of citizens' private data. By contrast, she
leaked military/diplomatic documents and footage. Not exactly a 1-to-1
parallel.

------
marcoperaza
Is there any evidence that Manning was able to express herself at such a high
level before becoming a public figure? I can't help but suspect that someone
is just using her name for their own political ends.

~~~
jlgaddis
I've no idea, to be honest, but as I was reading the article I was thinking
that this could potentially be a nice way to provide for herself financially:
writing articles for the media, making appearances/speeches at
events/conferences, etc.

With a criminal record and fresh out of prison, she's not going to have many
(decent) opportunities available to her so perhaps this is one way to make the
most of her situation and/or experiences. I'm sure there are plenty of
organizations that would pay for this.

~~~
tanderson92
> With a criminal record and fresh out of prison, she's not going to have many
> (decent) opportunities available to her...

She was named as a visiting fellow at the Harvard Institute of Politics,
literally today.

[https://twitter.com/HarvardIOP/status/907977942269091840](https://twitter.com/HarvardIOP/status/907977942269091840)

~~~
jlgaddis
I'm two days late in responding but apparently they have since _rescinded_ her
fellowship.

------
nether
We need to blast off into space sooner than later to escape this tyranny. We
must declare independence.

~~~
kelnos
What makes you think we won't just establish similar tyranny on another
planet?

~~~
ColanR
It worked well enough for the US, at least for a while.

~~~
kelnos
And that's the rub: it does work for _a while_ , but humans inevitably
organize into power structures, and those structures always end up making
decisions at the expense of the minority.

------
Shivetya
We have had thought crime and precrime punitive actions and enforcement
already through the invention of hate crimes. we already have guilt by
association, including both involuntary and voluntary association. how that is
going to be any different in a computerized world is beyond me, other than it
might become less influenced by people which removes power from those
exploiting it.

~~~
lovich
It's not less influenced by people it's just inflexible now. People are still
writing the software. You get preconceived notions baked into something that
no longer even has the option to compensate. For example cameras with facial
recognition that thinks east Asians are blinking all the time[1]. This wasn't
malicious, it wasn't even intentional, but no one can go "Oh the system is
making a mistake" and fix it. Yes software can be updated, but how often do
small minorities get software updated for them unless they are paying more
than everyone else?

[1][http://content.time.com/time/business/article/0,8599,1954643...](http://content.time.com/time/business/article/0,8599,1954643,00.html)

