
Apple is sharing your facial wireframe with apps - lisper
https://www.washingtonpost.com/news/the-switch/wp/2017/11/30/apple-is-sharing-your-face-with-apps-thats-a-new-privacy-worry/
======
yeldarb
I've been playing with the TrueDepth Camera APIs on the iPhone X. Some things
I've noticed:

1) The ARKit "Face Mesh" seems to be a standard model that is scaled and
skewed to fit your face (for example, it ignores glasses, still works if you
put your hand in front of your face, etc). It is _not_ a 3D scan.

2) The "TrueDepth" data is not really all that granular. It seems similar to
the depth map you get from the rear-facing cameras on the "plus" sized models.
Here's what the sensor data spits out:
[https://twitter.com/braddwyer/status/930682879977361408](https://twitter.com/braddwyer/status/930682879977361408)

3) Apple is really good at marketing. It's been shown that, even if you cover
the TrueDepth camera, features that "require" it still work fine (including
Animoji and the apps that I've been developing using the front-facing ARKit
APIs).

3.1) The lack of Animoji and front-facing ARKit seems to be a software
limitation made for business reasons rather than a hardware limitation. See:
Google's Pixel 2 portrait mode photos done using a single front-facing camera
that have stacked up well against the ones from the iPhone X.

4) The scary part, which is vast dystopian databases of facial fingerprints is
already being done with normal photographs. The depth data is not needed.

I agree with the author that the privacy implications of all-encompassing
databases could be scary. But I disagree that this has anything to do with the
iPhone X or its TrueDepth camera.

~~~
Osmium
> 2) The "TrueDepth" data is not really all that granular. It seems similar to
> the depth map you get from the rear-facing cameras on the "plus" sized
> models. Here's what the sensor data spits out:
> [https://twitter.com/braddwyer/status/930682879977361408](https://twitter.com/braddwyer/status/930682879977361408)

As @braddwyer himself notes, you can probably get a _much_ better mesh
integrating over time. It depends how long it takes to capture a single frame,
but I imagine that's not long, so getting an order of magnitude improvement is
probably quite easy.

> 4) The scary part, which is vast dystopian databases of facial fingerprints
> is already being done with normal photographs. The depth data is not needed.

And yes ... after all, humans are quite capable of identifying people with
high accuracy from 2D photographs. Depth maps are not required for there to be
serious privacy issues with such databases.

~~~
yeldarb
@braddwyer is me :)

It gives you about 15 fps of depth data

~~~
Osmium
Haha, cool! Hi :)

Given the small size of the laser projector, I imagine natural movement from
the phone being hand-held would result in significant displacement of the
projected dots over a 1s interval? Have you tried integrating the 15 frames to
see what it looks like?

~~~
yeldarb
I haven't yet.

We submitted a game about 3 weeks ago using front-facing ARKit as its core
game mechanic and it hasn't been approved by Apple yet.

I'm waiting to see if they're going to allow us to use the new technology in
novel ways or not before I invest a lot more time in it.

------
adpirz
I'm honestly less worried about Apple than others. They've at least made some
measures to prove that they are willing to go some distance to protect
privacy, even losing ground to competitors in voice recognition.

Now thinking about co's like Facebook that not only has access to far more
imagery of faces tied to sentiments and moments but have shown time and time
again that privacy is a secondary concern AND that they're willing to use any
and all that data to actively pursue vulnerable populations [1], I get quite
nervous.

[1]
[https://www.theguardian.com/technology/2017/may/01/facebook-...](https://www.theguardian.com/technology/2017/may/01/facebook-
advertising-data-insecure-teens)

~~~
lisper
> I'm honestly less worried about Apple than others.

That's a pretty low bar. Yes, Apple at least gives lip service to security
which other companies don't even bother to do, but Apple has had some pretty
major security screwups lately (three in the last few weeks). You might be
less worried about Apple than the competition, but you'd do well to be
somewhat worried about them nonetheless.

~~~
thisacctforreal
Two other issues I'm worried about with iOS:

\- MAC address tracking (RTS packets thrwart MAC randomization[0], not
immediately clear if WiFi is fully off[1])

\- All-or-nothing access to photos on the phone (append-only for apps that
request it)

Discord in particular is bad for the photo permission, as I don't trust
China's Tencent with access to all the photos on my phone (which can include
lots of frequent location information!), but pasting images doesn't work in
the app.

I had assumed the "Photos" permission meant "permission to prompt this
dialogue" and "permission to save to Camera Roll".

[0]
[https://arxiv.org/pdf/1703.02874v1.pdf](https://arxiv.org/pdf/1703.02874v1.pdf)
[1] [https://support.apple.com/en-us/HT208086](https://support.apple.com/en-
us/HT208086)

~~~
saagarjha
Does Tencent not have an extension that pops up in the sharing sheet for
images?

~~~
thisacctforreal
Just checked again: not for Discord. No idea about other apps owned by
Tencent.

It's notable Discord was developed as an American startup, and it's not clear
what Tencent's involvement is. Regardless, for me it's too much access for a
chat app to have in exchange for the convenince of sharing a photo from my
phone.

~~~
zionic
A better solution would be for Apple to provide a decent photo picker that
functions at the system level, and require a separate (special) permission to
access all photos with the appropriate warning if that app needs a fancy dancy
photo picker.

Why do I need to give snapchat access to all photos ever just to post from my
camera roll?

~~~
jacobwil
It already does work like this in iOS 11. Apps can present the System photo
picker to you and receive only your selected photo while having their Photos
access set to "Never".

If you want to try it out install the Wire messenger (if you make the account
with a web browser you don't need to provide a phone number), and try to
attach a photo but deny Photo library permissions. (Here's the buttons to
press: [https://imgur.com/a/gc5Iq](https://imgur.com/a/gc5Iq)). Other apps
work this way on iOS 11 but this is the one that came to mind.

------
dmart
You know how some F2P games will allow you to watch video ads in order to e.g.
earn extra in-game currency, activate a point multiplier, etc?

As it stands, you can start one of these ads and then turn your phone upside
down, or look away. How long until an advertising provider makes use of the
attention API and makes it so that you _can 't look away?_ Seems bleak.

~~~
bcyn
"15 Million Credits" from Black Mirror vibes there.

~~~
bluetidepro
This came to mind to me, too. haha

------
Bud
Clickbait headline. Apple isn't doing what the headline implies. The
headline's hype also does not remotely match the contents of the article.
Which leads me to discredit the article in general and look for a more
reliable source on what is actually happening.

------
natch
Let’s not jump to conclusions here based on assumptions and ignorance. What is
likely being shared is a generic wireframe with pose and expression
information. Not a face fingerprint as some are breathlessly calling it.

~~~
zeep
a precise wireframe could be as precise as a fingerprint... If it has a very
high mesh count, it is just a high resolution 3D model of your face... much
more information then what is required to face-id you

look at this model for example, it has medium amount of mesh:
[http://image.shutterstock.com/display_pic_with_logo/279553/1...](http://image.shutterstock.com/display_pic_with_logo/279553/108981035/stock-
photo-high-resolution-concept-or-conceptual-d-wireframe-human-male-head-
isolated-on-black-background-as-108981035.jpg)

~~~
adjkant
Not sure why this is being downed. I haven't looked into the details of this
case, but the more detailed the wireframe mesh the faces are being fit to, the
more possibilities there are, and the lower the likelihood of collisions
between two people. Think of it as a hashmap with the number of buckets being
determined by the mesh quality. At a certain point, a few collisions won't
hinder apps from capitalizing on the information.

~~~
StanislavPetrov
Apple is very popular among the HN crowd. Any comment critical of Apple or
skeptical of their intentions will be immediately rejected by a substantial
number of readers.

------
tinus_hn
So you mean to tell me that an app that gets camera permission can get at the
things seen by the camera? Oh no!

------
WhiteNoiz3
Isn't this a bit like saying that Apple is sharing your photograph with apps
[that use the camera]?

------
TaylorAlexander
What’s interesting is that on the previous security method - fingerprint - no
data was shared with apps.

If your face is now a security feature and the data is being shared with apps,
that sounds like a security leak.

If an app collects your face data, can that data be subpoenaed by a court to
attempt to unlock your phone? If a court could work with an outside party to
use subpoenaed face data to 3D print your face, could they try to use that to
unlock your phone?

I’m satisfied with the fingerprint scanner on my phone. I don’t feel like I
need the change in tech. I understand if you’re really concerned about
security use a passcode only, but it’s still true that the new face unlock is
“differently secure”.

~~~
baddox
The thing is, I don’t think the 3D scanner really introduces that much
functionality beyond what could be done with any normal front-facing camera
and software. I’d bet that a few seconds of normal movement would provide
enough parallax to build a decent 3D model. In fact, the Apple APIs may not
even use the 3D scanner, given that Animojis apparently work with the scanner
covered up (and don’t work in the dark).

~~~
djrogers
>the Apple APIs may not even use the 3D scanner, given that Animojis
apparently work with the scanner covered up (and don’t work in the dark).

Animoji uses all of the front sensors - RGB and TrueDepth.

From [https://www.imore.com/yes-animoji-uses-truedepth-camera-
syst...](https://www.imore.com/yes-animoji-uses-truedepth-camera-system-
iphone-x):

> the TrueDepth camera system captures a crude depth mask with the IR system
> and then, in part using the Neural Engine Block on the A11 Bionic processor,
> persistently tracks and matches facial movement and expressions with the RGB
> camera.

------
nerdponx
The scariest part about this tech is not what _my_ phone will do with my face,
but what _other people 's_ phones will do with my face. Facebook, Snapchat,
and Instagram were bad enough. Am I going to have to start wearing a mask in
public just so I don't have my face tracked, sold to the highest bidder, and
left in unsecured databases for hackers to obtain?

------
mortenjorck
I can appreciate the desire to limit the proliferation of permission dialogs,
but this seems like a case where the implications are different enough to
warrant a separate one from the camera dialog.

This has some similarities to the location permission dialog, which was
updated in iOS 11 to differentiate between "allow always" and "allow while
using the app." Perhaps the camera permission dialog could be updated to
"allow continuous access" and "allow when taking photos."

------
bwang29
The problem in here is more or less awareness of sharing or how to prevent
accidental sharing of face data. I can see sharing to some photo editing app
in anonymized fashion being useful for certain things and better than say,
using a flat photo from Instagram or Facebook, but sharing faces on social
media is a lot more intentional than a small and hidden agree to share with
app button.

I think a better solution is when app is specifically requesting face data,
there is a 2-3 seconds mandatory decision time with the default option to be
off and prompt this decision in a different UI from the classic permission
request dialogue. In this way the user knows the request is different And is
given time to actually decide before accidentally tapping agree.

------
walterbell
This article has an image comparing iPhone X mesh resolution to other 3D
depth-sensing cameras, [https://www.linkedin.com/pulse/warby-parker-should-
you-worry...](https://www.linkedin.com/pulse/warby-parker-should-you-worry-
newest-app-interview-tomas-petrikaitis/)

------
peruzin
I mean, didn't they demo the wireframe masks for Snapchat during the keynote?
Didn't that imply that apps can see your facial wireframe?

------
polock
WHaaaaat? Does Apple explain details about this? When the moment they share
those data to other apps?

------
fjsolwmv
Is it controlled by app permissions?

~~~
jchb
Yes, same as the camera record permission (NSCameraUsageDescription).

------
Sir_Cmpwn
This is 100% likely to be sent to advertising companies who will 100% attempt
to use it to track you near their billboards et al.

~~~
acqq
And to develop their own new "fingerprints" and "big data" bases about who
"owns the eyeballs".

------
arca_vorago
Notice the deafening silence.

~~~
arbie
As I understand it, apps will have access to emotions and other abstracted
info about the user, but not the "face" as a reproducible element. This will
be interesting for engagement analysis.

~~~
AlexandrB
> This will be interesting for engagement analysis.

Of course it will, which is why this is another very troubling development in
phone software. It's bad enough that most (all?) interactions are logged and
data mined in a modern app. Soon emotional data will be too.

------
teknologist
Time to switch to Android.

~~~
k3a
Situation is quite bad for all smartphones.

But for Android there is at least f-droid.org repository made of apps
automatically compiled from the source code. I prefer using these apps. And
the fact you can install what you want. In iOS you are locked into a censored
AppStore and have to pay to get a certificate to run your own app.

There is also a small chance for someone to develop an Android phone with
fully open software in the future. There is 0% chance to have an open iPhone.

~~~
sklinger
Not 100% true. On iOS you can install your own apps from Xcode for free, you
have to pay the $100/year developer fee to distribute it on the app store.

------
gaius
So let me get this straight: you can unlock your iPhone with your face and
Apple is giving away a free high-quality 3D scan of your face to anyone who
wants it, who may or may not also own a 3D printer?

~~~
briandear
You don’t have it straight. That isn’t what’s happening.

------
dejawu
Something alarming occurred to me: Apple _has_ to be saving all of the faces
of people who come into Apple stores and do the Face ID demo. There's no
reason for them not to do that, they'd be leaving good data on the table
otherwise.

~~~
eridius
On the contrary, Apple explicitly would never in a million years want to do
that. That's a massive liability with no upside whatsoever.

Google and Facebook and other such companies treat personal data as an asset
and try to collect as much as they can.

Apple treats personal data as a liability and wants to have no more than they
need to operate.

------
reacharavindh
So instead of asking Apple API to authenticate a user via FaceID, apps are
going to obtain this data themselves and record it forever? I remember Steve
talking on stage about how Apple would always make the operating system retain
control of sensitive sensors and only give the result to user-approved
apps(asking for permission every time). That's all thrown out of the window
now?

~~~
djrogers
This is not FaceID data, nor is it a replacement for it - the data available
to apps is less granular than that used by FaceID, and is designed to give app
developers the ability to create things like Animoji or any other creative
uses of the 3D sensor data.

>I remember Steve talking on stage about how Apple would always make the
operating system retain control of sensitive sensors and only give the result
to user-approved apps(asking for permission every time). That's all thrown out
of the window now?

Why would you think that? Apps do request permission for this, and it can be
revoked.

------
LeicaLatte
What is this 'facial wireframe' this Jeff Bezos owned newspaper is speaking
about? Looks invented just to write this article.

~~~
yeldarb
It's unclear whether he's talking specifically about the ARSCNFaceGeometry[1]
or the ARFrame.capturedDepthData[2] or some combination of both.

[1]
[https://developer.apple.com/documentation/arkit/arscnfacegeo...](https://developer.apple.com/documentation/arkit/arscnfacegeometry)

[2]
[https://developer.apple.com/documentation/arkit/arframe/2928...](https://developer.apple.com/documentation/arkit/arframe/2928208-captureddepthdata)

------
pmontra
The FaceID camera is infrared and can see in the dark [1].

According to the Post "Once you give it permission, an active app keeps on
having access to your face until you delete it or dig into advanced settings.
There’s no option that says, “Just for the next five minutes.”"

So a random developer can (kind of) look at us when we turn the light off. Not
only that, those cameras work also in the sunlight. You can go on YouTube and
search "clothes transparent to infrared". Not much, especially considering
that the app is not getting a real image, but still somebody will be
uncomfortable with that.

However there also legit and interesting applications for infrared cameras and
they've been on sale for a long time. What's changing is the number of those
cameras that are getting around.

[1] [https://www.macrumors.com/2017/09/13/how-iphone-x-face-id-
wo...](https://www.macrumors.com/2017/09/13/how-iphone-x-face-id-works/)

~~~
_puk
"Once you give it permission, an active app keeps on having access to your
face”

Isn’t the definition of an active app, one that is open and in the foreground?

The minute the phone sleeps, or the app is put into the background, access
becomes more at Apple’s discretion.

I’d expect an app running in the foreground to be able to keep using the
camera as long as needed..

Or is your point more: allowing once allows it every time the app opens, with
no option to repeatedly prompt?

~~~
pmontra
I have no direct experience of iOs but according to the post it's as you
wrote. It could be an app that needs the camera once for some sensible reason,
then it can use it forever when it's active even if it wouldn't need it from
the point of view of the user.

~~~
jpttsn
Users might notice the battery penalty.

------
mtgx
Tell me again how FaceID will be "so much more secure" than TouchID - when
Apple itself is sharing those 3D facial profiles with third-party vendors (and
governments).

In _practice_ , fingerprint readers should still remain _significantly_ more
secure than any facial recognition technology.

~~~
djrogers
Apps get a coarse depth map - this _cannot_ be used to replicate your face in
perfect 3D, nor can it be used to bypass FaceID.

~~~
wuliwong
I'm not a security expert but this seems like are giving out a coarse
approximation of my username (that has no password protection). I would think
it is significantly easier to break into FaceID starting with the coarse depth
map than no depth map at all. But again, I could just be out of my depth on
this topic and the difference isn't that big of a deal?

~~~
yeldarb
It's possible but I 3D printed the face mesh from their API and it didn't work
to unlock Face ID on my phone.

It did a pretty good job of scanning my face though:
[https://twitter.com/braddwyer/status/930594896523567104](https://twitter.com/braddwyer/status/930594896523567104)

