
Apple suspends Siri response grading in response to privacy concerns - Doubleguitars
https://techcrunch.com/2019/08/01/apple-suspends-siri-response-grading-in-response-to-privacy-concerns/
======
probably_wrong
So far, most of the comments here are a good example of both why this happened
and why it will happen again.

When people complained that putting microphones in your home 24/7 was creepy,
the response was "that's nonsense, Siri is not sending anything unless you
specifically ask for it". When it turned out that it _does_ send things you
didn't intend, the response was "well, it's not like anyone is listening
anyway". Now that we know someone _is_ listening, the response is "it's no big
deal, because they don't care and ultimately is good for you".

Just because most people don't understand the implications of what they agree
to, it doesn't mean they don't care. And the levity with which the tech
community disregards other people's privacy concerns seems to me one of the
reasons why everything everywhere is spying on us nowadays.

We really need mandatory ethics courses in CS.

~~~
abhisuri97
Minor point: While I do agree that ethics course should be a part of CS
curricula, wouldn’t managers be the ones who likely pushed for the pattern
you’re remarking upon? (Ie ethics classes would be more useful for them?)

~~~
IfOnlyYouKnew
Tangentially related: The Technical University of Berlin is considered to have
one of Germany's best departments of philosophy.

The reason: after World War 2, the university was located in the British-
controlled sector, and their administration made philosophy mandatory for all
students.

Because never again should young people be given the power of an
engineering/science education without any regard to the ethical implications
of the possibilities.

~~~
ekianjo
> have one of Germany's best departments of philosophy.

So what? Philosophy has barely anything to do with ethics. You can study
philosophers that can argue for complete opposite sides of the same point at
hand, so it does not give you any indication as to what is right or wrong.

And Germany has a pretty poor record so far in terms of privacy
protections/provisions, so I am not sure there is any indication that post-WW2
or post-RDA experiences have had much impact.

~~~
sooheon
An education should not hand you a canned answer to what is "right" or
"wrong", it should do almost the opposite. Learning to evaluate well founded
opposing arguments is certainly more useful than being force fed moralist
dogma du jour.

~~~
ekianjo
You can't evaluate arguments in a vacuum. You need a value system to do that,
and Philosophy doesn't give you any, it's merely the art of debating.

~~~
sooheon
Calling it "merely the art of debating" implies it lacks actual substance. I
think philosophy concerns itself with _what_ you debate about, not _how_ you
do it.

Besides, nothing outside of dogma can "give you" a value system, but
philosophy does present candidates. It's up to you to choose your own, the
hope being that this is better than unquestioned inheritance of values.

------
shalmanese
People have really bad intuitions around privacy. The predominant mental model
is that there is a single, motivated adversary who is uniquely interested in
attacking them. In reality, privacy almost always involves large, impersonal
systems ingesting masses of data to either increase profits or commit crimes.

It may sound bad that some random person is hearing your sex sounds but that
person has heard a thousand other equally anonymous sex sounds and truly could
not give less of a shit that the sound is of someone having sex. Similarly
with leaking financial information, anyone capable & willing to defraud people
based on this information has much better ways of generating ROI than hoping
on the one snippet of information in a million that could lead to a breach.

It's similar to that time it was revealed a bunch of iOS apps were screen
recording users using the app. Having worked at companies that collected that
data, the _only_ conceivable use we had for it was to help diagnose bugs for
individual users having issues with the app. Every other question we wanted
answered, it was much easier to use aggregated data collated specifically for
that purpose. A bunch of videos of people tapping the screen was less than
useless for that. And yet when I told people about that, in general, they
refused to believe me and believed there must be some super secret nefarious
reason we were collecting such "invasive" information.

~~~
calcifer
> People have really bad intuitions around privacy.

> Having worked at companies that collected that data [...]

No, you just have a different personal definition of privacy. The fact that
_you_ think _your_ use of _my_ data is innocuous doesn't mean you are not
violating my privacy. If your use was really, trully unobjectionable you'd be
fine with getting clear and explicit consent.

Somehow, I don't think an honest consent prompt like _" Can our app silently
record your screen and send it to us?"_ would get you the results you want.

~~~
joshuamorton
> If your use was really, trully unobjectionable you'd be fine with getting
> clear and explicit consent.

This ignores human nature: requiring people to make a decision, any decision,
is a friction point that will lose users. Something like "we'd like to
increment a counter when our app crashes, we won't log anything else than that
single bit".

So first of all, you'll have people who will claim that you could be lying,
and they'll opt out, or not install your app, or whatever. Fine. But now all
the people who truly didn't and don't actually care will have to make a
decision. And making choices takes effort. So some people will just refuse to
make a choice, and you'll lose users.

Consider the other side of this: I'm from the US but currently in Europe. GDPR
requires explicit consent for the various trackers, it has defaults. I don't
change them. I don't want to think about them. I don't care enough. And it's
just effort I have to expend before I actually get to the stuff I'm interested
in. Like a government mandated paywall (think the "we use cookies..." but more
extreme, with checkboxes to decide which cookies they can store).

Even as someone who truly actually doesn't care, and is happy to, in the
abstract, give explicit consent to all the tracking these sites do, I don't
want to have to explicitly consent, because its work to do something I don't
care about.

I expect, though I'm by no means certain, that most people who aren't on HN
are in that bucket. And that means that requiring explicit consent for
otherwise innocuous things is just bad for business with little gain, and it
numbs people for explicit consent on actually important things.

~~~
calcifer
> requiring people to make a decision, any decision, is a friction point that
> will lose users

One could argue that's a good thing.

> you'll have people who will claim that you could be lying

That's not a claim, that's a fact. You _could_ very well be lying and I
wouldn't know as I don't have access to the internals of your company.

> But now all the people who truly didn't and don't actually care will have to
> make a decision. And making choices takes effort.

The horror!

> So some people will just refuse to make a choice, and you'll lose users.

Good, people shouldn't blindly sign up to services they don't understand just
because it's a new shiny thing.

> I don't change them. I don't want to think about them. I don't care enough.

Good for you. I'm also in Europe and I think GDPR is fine and I _do_ change
the cookie settings instead of just clicking through.

> And it's just effort I have to expend before I actually get to the stuff I'm
> interested in. Like a government mandated paywall

Music to my ears.

> requiring explicit consent for otherwise innocuous things is just bad for
> business with little gain

I couldn't care less about what's good or bad for a data collecting business,
I care about the people whose data will inevitably be leaked by said business
in a future breach.

~~~
joshuamorton
> One could argue that's a good thing.

Then you should make that argument. If you think it's a good thing, put in the
effort of saying why. Don't force me to imagine why it might be so.

> That's not a claim, that's a fact.

Sure. But asking for explicit consent for the innocuous thing: logging a bit
and nothing else, doesn't change the fact that the company could be lying.
Nothing is gained. If they're lying about logging the bit, they might be lying
if you opt out. Obviously it's far easier to figure out what is being sent
than what is being being logged serverside, but you'd still need to verify
that independently. If you don't trust the group providing the service, unless
its verifiably trustless (not communicating at all or client-side encrypted
with minimal side channels), you shouldn't use the service. Otherwise, you're
still trusting the service provider.

> The horror!

Yes. Making people expend mental energy on otherwise unimportant/irrelevant
decisions is anti-user.

> Good, people shouldn't blindly sign up to services they don't understand
> just because it's a new shiny thing.

My point is a lot of people do understand and just don't give a shit. Getting
in the way of those people is anti-user.

> Music to my ears.

Again, anti-user.

> I care about the people whose data will inevitably be leaked

I'm talking about innocuous things. Things where being leaked isn't a problem:
truly anonymous or aggregated data that can't be used to identify or tied back
to an individual user or group. You're saying users should still need to opt
in to those kinds of tracking. That's anti-user. It obfuscates actually
harmful tracking.

To phrase it another way, there's PII and not-PII. Are you suggesting that
tracking of non-PII need explicit user consent? Even if revealing non-PII, by
definition, can't be tied to an individual user?

>by said business in a future breach.

Recently, we've seen leaks of PII from CapitolOne and Equifax. Click boxes on
websites don't help when the data being leaked is banking information that
(for capone) has to be stored tied to an identity and can only be opted out of
by not using the products in question (credit cards, banks), or isn't gotten
from the user at all (equifax).

How does clicky-consent boxes on their websites help with that?

~~~
raxxorrax
> Making people expend mental energy on otherwise unimportant/irrelevant
> decisions is anti-user.

Then don't ask them and let them explicitly toggle the option to share
information. That isn't anti-user. Your privacy invasions are anti-user.

~~~
joshuamorton
> Then don't ask them and let them explicitly toggle the option to share
> information.

Is incrementing a counter if an app crashes a privacy invasion?

~~~
raxxorrax
Yes, because you get a bunch of meta information when any device of mine uses
my internet connection to connect to your servers to increase that counter. If
you save that information or not, you created a trail.

I expect from any app that it doesn't establish any connections per default to
be honest. You can have that as an option that I can enable if I feel like it.

~~~
joshuamorton
Ok, so what if the app is internet connected and has to communicate with some
central server? Think a non-p2p messaging app, or an app for a bank/<some
internet thing like youtube or facebook or reddit>?

~~~
raxxorrax
Then it is obviously allowed to do that for that specific purpose. If you want
your flag to be packaged into the data stream, it has to be some option that
says "send additional diagnostic data".

------
noodlesUK
I’m genuinely concerned about data privacy, but I’m kinda surprised about the
outcry about all of this. I guess it’s relatively common knowledge for hacker
news readers that all of these systems need to have human intervention to
improve, but I feel like all this could have been avoided by just explaining
that this was how the service was able to improve... I’m sure more steps could
be taken to anonymise data from the people doing the transcripts, but I feel
the best part of this is to improve the wake word detection to reduce the
false positive rate.

~~~
donkeyd
I mostly agree, however, there are some issues I'm foreseeing. I've done some
NLP work myself and have had access to very private information because of it.
The team we were working with were all highly educated and well paid. Because
of this, we were aware of the implications of the data we were working with,
so at one point, in one of our randomly selected data sets, we found data that
was from a public figure and removed it to avoid any possibility of a conflict
of interest.

Where my concern is, is that a lot of annotating work is now being done by low
wage, low education workers and because of the increasing demand of annotated
data sets this group is increasing. Also, because this work is increasingly
getting outsourced, there is less direct control of who is doing the work and
therefore the chance that a bad apple slips through is bigger. That's what's
scaring me going forward.

~~~
kmlx
i’d wager that in most cases “low wage, low education” workers have much
better morals than your “highly educated well paid” team.

~~~
usrusr
The problem isn't a single-value metric of morals, whatever that may be, the
problem is that one group is bored and not invested in the project while the
other has far more exciting things to do than wondering who they might be
listening to.

~~~
donkeyd
Thanks, that is exactly what concerns me.

------
anilakar
Protip: When you use Google's handwriting recognition, all the texts you write
are visible to people who have opted in to train the neural networks. I've
already received tons of names, phone numbers and even something that resemble
passwords.

~~~
sah2ed
Hearing about this for the first time.

Mind linking to more info on how people opt in to help train Google’s NN for
handwriting recognition?

My searches turned up this TC article (from 3 years ago):
[https://techcrunch.com/2016/08/29/googles-new-app-
crowdsourc...](https://techcrunch.com/2016/08/29/googles-new-app-crowdsource-
asks-users-to-help-with-translation-transcription-more/)

Thanks

~~~
gmemstr
From what I remember, the crowdsource app is exactly what is used for
community sources.

------
rammy1234
[https://techcrunch.com/2019/08/02/google-ordered-to-halt-
hum...](https://techcrunch.com/2019/08/02/google-ordered-to-halt-human-review-
of-voice-ai-recordings-over-privacy-risks/)

------
anbop
My guess is that this whole story arc was planned from Day 1. They knew they’d
need to do this, and they’d get caught, so they just tried to tag as much
stuff as they could before the shit hit the PR fan.

------
GoofballJones
So, Amazon, Google and now Apple have been "caught" doing this. Is Microsoft's
Cortana doing it also on it's voice commands?

Also, wouldn't it make sense for EVERYONE to just use their own users to gauge
if the voice is working as intended? Like, once in a while get a random "was
this correct?" from the people using it, instead of strangers determining it.

------
ksec
The biggest problem of all is _Apple_ , a company which constantly brags about
its privacy stand, is that doing this without explicitly telling its user.

One of the thing with Tim Cook's Apple is the smell of Hypocrisy. I am giving
Apple's the benefits of doubt, and they will need to explain this better.

------
jammygit
ugh, techcrunch seems to automatically swap pages on you if you scroll down.
What an awful ui decision

------
fmajid
Too little, too late. Siri was already disabled on my Macs, I now disabled it
on all my iOS devices.

~~~
jhayward
I just don't enable 'hey Siri'. I use Siri but summon it by mashing the power
button. Siri is useful and I feel comfortable with its explicitly summoned
mode.

------
throw03172019
I actually think they should collect this data. Siri doesn’t understand basic
things from me.

~~~
thepangolino
Can’t it be opt-in?

~~~
bb123
> Additionally, as part of a future software update, users will have the
> ability to choose to participate in grading

~~~
6gvONxR4sf7o
That could mean opt-out.

------
giancarlostoro
On the one hand it sucks because clearly the quality might be lowered. On the
other hand if Apple invests on Siri doing its thing mostly on the device like
Mycroft does that would be worthwhile.

------
drderidder
End users should be the ones grading Siri's responses anyway.

------
bredren
Can anyone give insight into how grading is specifically used? Are the results
used for ML training?

~~~
bilbo0s
Well, it isn't anymore apparently.

------
6gvONxR4sf7o
How about we start treating these sorts of freeform data with the highest
security and privacy they deserve? If it might contain HIPAA data, treat it as
you would HIPAA data. If it might contain military secrets, treat it as you
would under top secret clearance. If it might contain personal data, treat it
as you would under GDPR. When it's always on for a huge portion of the
population, treat it like it's likely enough to contain data that should be
held under a really high standard. Don't treat it under the average case
standard.

If your users are usually saying "Hello, i am unhappy I spent two hours on
hold" but a few thousand data points say "hello, my name is john doe. I have
HIV, I live at 123 Main Street and my social security number is 987654321"
treat the whole dataset as the latter case. Maybe we should even pass
regulations along those lines.

------
kmlx
yes, make something that doesn’t work correctly even worse. at this point why
have siri in the first place? and how long will it take for them to fix it? 6
months?

------
vbezhenar
Siri is already useless when comparing to Google Assistant. And they want to
handicap themselves even more. Apple will lose AI fight to Google in the end.
Some people probably will value privacy over convenience, but that's certainly
not me or majority.

~~~
mrtksn
Are people really using these assistants for anything more than making the
device do something? Because in my experience Siri will set reminders, timers
etc. just fine and will find the song you are listening just fine and will
call someone just fine.

The other stuff feels like a gimmick to me or more like stuff that some
middle-level manager spawned to get a nice bonus.

Using a voice assistant "AI" usually feels like trying to make a really stupid
person do something for you. It will accomplish 90% of the thing that I am
trying to do for %75 of the times and then it will completely miss the point.
Okay, maybe with Google it is like %85 of the time but when it misses the
point still misses the point completely.

~~~
completelylegit
The day when you can just say “lights on, 50%” and the assistant in the next
room understands you don’t mean it is when we will have the right amount of
universal compatibility.

Until we can derive decisions based on who is speaking and where they ”are” -
I think it’ll always be clunky.

“Hey Alexa tell Hue to set light scene livingroom to 80%” is cool that it
exists for those who really need the solution but its not easier or cool for
sure but we’re so far from that.

~~~
lokedhs
I listen to the Twit podcast, which is usually very good. One thing they do,
however, is to stop themselves from saying the word Alexa, and if someone
accidentally does so, they beep it out.

Why? Because they don't want to set off any so-called smart speakers.

I'd argue that going so low as to change the way we speak just to accommodate
these devices is quite terrible, and if you have to watch what you say so that
a smart speaker don't misunderstand you, then it's not very smart.

Their listeners shouldn't complain that their hosts use the word Alexa. They
should complain to the vendors that their products are garbage.

I wouldn't use one of those even if they weren't garbage, but at least I
wouldn't be infuriated every time I hear someone on the podcast stop
themselves from saying the name of the product that's the topic of their
discussion.

~~~
FiddlerClamp
In our house we refer to Alexa as "Alesto" when we're discussing her.

