
A UI Experiment with the iPhone X’s Front-Facing Camera - dirtyaura
https://www.fastcodesign.com/90162217/a-wild-ui-experiment-with-the-iphone-xs-front-facing-camera
======
russellbeattie
The much maligned Amazon Fire Phone had _four_ specialized high-speed head
tracking cameras on the front of it, and made this effect the centerpiece of
it's UI. I worked on it, and it was very cool. The phone failed for lots of
reasons, but this particular "gimmick" actually held some promise. The example
clay-molding game which came with it actually showed how being able to
perceive the image in pseudo-3D did have a real application.

~~~
berdon
I worked on it too. You're right about the head tracking - it did work very
well. They had some great UX ideas centered around it too. It's unfortunate it
didn't pan out for those UX features alone.

~~~
subculture
Which were the great UX ideas? I think I missed them... The fact that you had
to move your head or phone just to see the time was bad. The 'peek' gesture
which required moving your head to the side and looking sidelong at the phone
to open a side navigation menu was equally lame.

~~~
russellbeattie
To be self critical, I think there was a lot of, "We've got this cool hammer,
let's make everything a nail" sort of thing going on where the perspective
stuff was grafted onto a lot of areas better left alone. That said, the tilt
scrolling on web pages was actually very nice to use (remember it was based on
the angle to your eyes, not the physical phone angle, so you could use it
lying down). And like I said, applications where it was helpful to glance
_around_ a 3D object, was quite compelling. But yeah, there were too many
areas where the effect was used where it probably didn't need to be.

------
drosan
Is it a joke? "Wild experiment"? "first to take the effect mainstream"? What
kind of bullshit is that?

The said effect and the head tracking principle was already used in the New
Nintendo 3DS consoles that was sold millions worldwide. Not only that - 3DS'
top screen has a true "parallax barrier" screen, means it displays a 3D
picture with depth and all.

~~~
sorrybut
Apple are the first to innovate all tech when one's entire worldview is
limited to the Apple product line.

~~~
digi_owl
And because Apple has the media/design world by the balls thanks to ingrained
habits, they have blinders on for the rest of the tech world.

------
cjmoran
I've always found these pseudo-3D effects (including the slight "floating-
icons" effect applied to iOS wallpapers) to be unconvincing in real life.

I suspect they look much better in demo videos like this because there's no
depth information in the objects surrounding the phone either -- the person's
hand, the floor/desk behind them, etc. This makes it much easier for one's
brain to map "pseudo-depth" onto the entire video.

In real life, you have depth perception for everything except the images on
your phone's screen, making a demo like this look "flat" even at its most
convincing. Real 3D requires two distinct visual inputs; this is a gimmick.

~~~
asadjb
The video does mention it works best with 1 eye closed. Maybe this is the
reason for that.

~~~
memebox3f
definitely the reason

------
ChrisRR
This reminds of the head tracking demo using a Wii remote made by Johnny Lee
years ago. [https://www.youtube.com/watch?v=Jd3-eiid-
Uw](https://www.youtube.com/watch?v=Jd3-eiid-Uw)

~~~
Lewton
I'm torn, there's literally a link to Johnny Lee's work in the article. So
your comment doesn't add anything.

But a lot of people read only the comments (or the comments before the
article) so your comment does add value

Is it a good HN comment or not? :D

~~~
bringtheaction
It’s an acceptable kind of HN comment. A link in the article is generally easy
to miss and probably others missed it too.

Admittedly the article did talk enough about the other thing that it’d be
pretty hard to miss if you read the whole thing, but I think we can forgive it
still. Perhaps the parent to your comment opened the page and saw the video
but didn’t bother to read the article. I think that’s fine too.

I always read comments first and in fact quite often I read the comments only.
As long as we don’t derail the discussion completely by starting a top-level
comment about how we perceive something based on the title alone I think it’s
fine. That includes it being fine to mention things that you are reminded of
by the title as long as you phrase it correctly. In my opinion of course.

------
d--b
This is neat.

I think what would be even neater is to use the head tracking from the notch
to compute a transformation of the back-side camera to simulate that you can
see through the iPhone. Of course this wouldn't work when the iPhone is too
close, or if the angle is too wide, but in other cases, this would make AR
really good.

~~~
220V_USKettle
I assume with two cameras on the back, it would be possible to create a stereo
video stream and implement a transformation as you described.

------
adamscybot
"The notch. Everyone hates it. And what did we get for it? A rooster that
sings karaoke. No thanks."

This isn't true. What we got for it is Face ID and the consequential removal
of the home button which then allowed for the frameless design.

Not saying it's ideal. But the author has missed the point slightly.

~~~
joshstrange
Yeah, everyone hates it so much that a ton of android phones being released
this year have their own notch... It really does just fade away for me, I
don't notice it anymore

~~~
TremendousJudge
and the essential phone, that had a notch before it was cool

------
jordache
Yo mark willson of co.design. Were you around 10 years ago?

Let's tame the article title a bit huh?

[https://www.youtube.com/watch?v=Jd3-eiid-
Uw](https://www.youtube.com/watch?v=Jd3-eiid-Uw)

~~~
giarc
He talked about Johnny in the article.

>Norrby is far from the first person to play with trompe-l’oeil (also called
“parallax”) in the modern age. Johnny Lee, now at Google working on augmented
reality, first made his mark by turning a stock television and a Nintendo
Wiimote into a wild 3D game.

~~~
jordache
oh ok.. so he should know it's not 'wild'.

annoying bait title.

------
Shivetya
Very cool but I do have to laugh at his opening remark, the notch and how
everyone hates it. If anything screams that Android phone makers have no
imagination it is the notch, suddenly its on their phones.

I do think the notch is bad UI but it can be worked around. I would have more
easily accepted a narrower band of black across the top and bottom but Apple
isn't losing sales because the rest of the phone's tech is that good.

I do look forward to the day when the components are barely distinguishable
from the display.

~~~
annabellish
_Some_ android phone manufacturers are adding pointless notches, but it's a
big market. You'll find some android manufacturer doing basically anything.

Personally I find Samsung's current designs prettier than the iPhone X, though
not by much - thing's gorgeous.

------
peteretep
Still not available on the app store. Are there _any_ apps actually taking
advantage of this yet, other than the animated emoji? Kinda bummed I've not
had anything nice to play with yet

~~~
John_KZ
>Are there _any_ apps actually taking advantage of this yet

Yes, apple is harvesting your high-quality biometrics using this. 5 years down
the line someone will leak the data and your face will end up on some weird
porno series.

~~~
aluhut
Critical Apple comments are risky here ;)

~~~
macintux
Stupid ones are.

~~~
aluhut
What is stupid about expecting this highly personal data to leak or be misused
these days...macintux...?

~~~
macintux
The strong implication of the original comment was that Apple was centrally
collecting biometrics and would eventually sell it off or leak it.

The biometrics are stored on each iPhone, in a secure enclave with severely
limited access, and they're only useful for identifying (slowly) someone at
close range.

So, yes, theoretically in the future someone could hack all iPhones everywhere
using one of the most sophisticated attacks ever, and collect data that might
conceivably be useful for high resolution cameras to identify faces at close
range. Maybe. Although Face ID uses infrared, so would the high resolution
cameras need to have that, too?

Anyway, that's why I felt it was a stupid comment.

~~~
aluhut
The bio-metrics shouldn't be collected in the first place. A simple patch on
this closed system would allow who knows whom to collect the data. Just
considering the current political situation in this companies home country
it's a unreasonable risk. Considering their willingness to cooperate with
dictatorships and similar ads to this. Fear not stupid at all.

Considering the fact that this exists because a) Apple was unable to come up
with any real innovation to sell the new product and b) people seem to be
really to lazy to type a number and rather give up that information.

Data Minimization, Data Avoidance are a thing.

> "using one of the most sophisticated attacks ever"

As if we haven't heard that speech before...

~~~
macintux
Users worried about their privacy can always choose not to use it.

The more likely attack vector is simply capturing scenes through the front
camera. That would give you, most of the time, an image of the face of the
user, not just depth mapping information of questionable value.

But really, nearly everyone shares images of themselves online, so even that's
of dubious value.

I'd challenge you to find any security expert who agrees that Face ID _as
Apple implements it_ can realistically result in useful biometric data being
leaked to a hostile party. Apple supplies whitepapers documenting the secure
enclave, I imagine there's one for Face ID.

Now, the question of whether Face ID is secure enough for any given user's
needs as a local authentication is a perfectly valid question, and clearly for
some users the answer is no. But, again, it's optional, and that's not at all
the threat under discussion.

~~~
aluhut
> Users worried about their privacy can always choose not to use it.

You make it look like it's easy for the normal Apple user to switch to
Android. In fact it's quite the opposite and the whole situation is even worse
because most of them won't even be aware of the dangers. The major reasons for
people I know to chose an IPhone over an Android is the "ease of use"
(resulting from the fact that their first smart phone was an IPhone already
and they never tried anything different) and because they are "so confused
with all the options/apps/general possibilities on Android". Those are the
people who need to be especially protected. They are caged within a locked
environment of a single US company. This alone should make you think.

> But really, nearly everyone shares images of themselves online, so even
> that's of dubious value.

This sentence together with this high tech approach you demonstrate on the
rest of the comment is mind-boggling. As it's the most common approach of
companies/individuals to abolish digital privacy all together. The old version
of it was "I'm not afraid of X because I have nothing to hide". Horrifying,
but now I understand where your attitude comes from.

Being born in a oppressive state, this is where I would actually use the word
"stupid".

> I'd challenge you to find any security expert who agrees that Face ID as
> Apple implements it can realistically...

As I've wrote above. It may be that FaceID is not a big deal right now. We
don't know it for sure since it's all locked down but we assume it. There is
however still the ARKit and all those APIs using those depth/facial mapping
capabilities. Those becoming the new standard for popular apps is just a
matter of time and since you've already given rights to use the camera, those
features will be (or are already?) a nice extra. So you see...we don't even
need to reach out to possible changes from the paranoid government governing
Apple and their data under and awaiting some patch to allow the access to
FaceID data. It's far more accessible.

I wonder, would you allow your phone to take a drop of blood for
authentication or where does your privacy actually start?

~~~
macintux
> You make it look like it's easy for the normal Apple user to switch to
> Android.

No, I'm saying Face ID is not mandatory on an iPhone X. You can use a
passcode.

Nor am I saying privacy should be sacrificed on the altar of technology. I do
my best to stay away from Google, and I try not to let Facebook know any more
about me than necessary (and every day I contemplate ditching it, but there
are a few important reasons to stick around).

There are plenty of ways to do biometric security wrong from a privacy
standpoint. I trust Apple to do an earnest job of doing it right, because they
have positive incentives to work for their user base rather than being a data
collection/ad selling company.

And if biometrics aren't where you wish to place your faith, you can simply
not enable the feature.

~~~
aluhut
Would opting out of FaceID also lock down the feature completely for the APIs?

FYI: you can have an Android phone complete without a single google app or the
google app store. LineageOS is the most popular alternative OS. There also
other stores you can put on your phone. Like F-Droid, which hosts open source
apps.

~~~
macintux
> Would opting out of FaceID also lock down the feature completely for the
> APIs?

Biometrics, yes. AR-based depth mapping, no.

~~~
aluhut
Really? Let's look in the App Store Guidelines:

[https://developer.apple.com/app-
store/review/guidelines/#dat...](https://developer.apple.com/app-
store/review/guidelines/#data-use-and-sharing)

5.1.2 Data Use and Sharing

    
    
        (i) You may not attempt, facilitate, or encourage others to identify anonymous users or reconstruct user profiles based on data collected from depth and/or facial mapping tools (e.g. ARKit, Camera APIs, or Photo APIs)
    

\-----

Let's wrap this up here.

\- A user issues his fears based on a technology that is the topic here

\- he gets downvoted into oblivion but no comment follows

\- I trigger a comment by stating the obvious behavior prevalent on every
single article posted here that may be or even is critical towards Apple

\- you declare the users comment stupid based on your assumption that a single
software use of the general feature may not be misused. Even though you can't
know that because we are talking about a closed system and the APIs allow that
without a possibility to opt-out (if you have already granted general camera
permissions).

\- you further state that users don't need privacy either way because they
gave it all away. Which is actually the only really stupid statement in this
discussion here

\- after all that you even go so far damning another US company based on
actually nothing. A company that allowed the world to develop their own open
source operating system and app world after you've done everything to protect
a company that provides you with a system you actually know only what they
allow you to know about.

macintux, I couldn't have wished for more to demonstrate what is wrong here.
There is a quasi cult behavior in the Apple fan base turning people into
marketing machines ready to drop everything to protect the brand while
condemning everybody else. You owe the guy an upvote. I don't care.

~~~
macintux
> I couldn't have wished for more to demonstrate what is wrong here

Then I am glad to have been of service.

~~~
aluhut
I like how you ignored everything else ;)

------
oceanswave
What’s the difference between this and the plethora of AR games and apps that
‘trick your mind to believing there’s a whole world beyond the screen’

Using face positioning vs the accelerometer is interesting, but without seeing
it in action, is it that much more compelling?

------
gambiting
>>The notch. Everyone hates it.

These broad statements irritate me. I don't hate it - I actually think it
integrates super well with the UI and makes the X look unique among the sea of
identically looking glass slabs. And I don't even own an iphone.

~~~
zwily
I don’t know any iPhone X owner that hates the notch. That opening bugged me
and I stopped reading there. :)

~~~
overcast
When you spend $1000+ on a phone, you'll find any reason to justify its
faults. If a person had a choice between the notch, and no notch, with the
same functionality. I'm doubtful anyone would choose the notch for aesthetic
reasons.

~~~
gambiting
I disagree. Think about it less as a notch in the screen, but as an extra
screen around the notch. I'm looking at my phone and it has the camera and
speaker above the screen, but the sides are just empty black space. If I
could, I would absolutely want to have extra screen there, to display the time
or notifications. So yes, I would absolutely pick a phone with a notch over a
phone without it.

~~~
overcast
The assumption was that the screen real estate would be equal, minus the
notch.

------
neelkadia
This is very similar to Johnny Lee's initial exploration with Wii...
[http://johnnylee.net/projects/wii/](http://johnnylee.net/projects/wii/)

------
sundvor
As much as I don't like the iPhone Notch and all that, the video in the
article was quite cool, appeared very performant and a great programming
showcase. So kudos to the Dev for making this.

------
220V_USKettle
How difficult is it to use/implement as a 3D scanner (3d modeling)?

~~~
soylentcola
Haven't messed with the depth sensor on an iPhone yet, but you could do it
with Kinects and other depth cams like the Intel Realsense series.

In my experiments with those, the main drawback was ergonomics (device of
varying size depending on the sensor, connected to a computer via at least one
cable as you moved it around the thing you were scanning).

On a phone or other standalone/self-contained bit of hardware, I'd imagine it
makes things easier.

~~~
mcphage
> Haven't messed with the depth sensor on an iPhone yet, but you could do it
> with Kinects and other depth cams

Well, if you’ve used a Kinect, then you’ve used the iPhone’s tech, because
Apple bought PrimeSense, which developed the initial Kinect for Microsoft.

------
mabedan
I'm pretty sure the identical effect can be (and has been) produced by the
combination of normal front facing camera, accelerometer and gyroscope.

~~~
baddox
The only required piece of information to render the scene is the position of
the eye relative to the phone, which can be represented as an x and y
coordinate relative to the rectangular frame captured by the front-facing
camera. The accelerometer and gyroscope shouldn’t be needed, except perhaps to
smooth out movement of the phone in the user’s hand if the camera’s frame rate
is too low.

My guess is that the only reason this demo is relying on the iPhone X is that
Apple provides a facetracking API, so there’s no need to write any code or use
external libraries to do eyetracking with a normal front facing camera.

~~~
rocqua
To really get a nice 3D effect you also need distance of the eyes from the
phone. Based on the video in the article, this demo actually uses this depth
information. You can't get this depth info easily from a camera. I can
certainly see how the more advanced optics in the iPhone X really help here.

~~~
baddox
I suspect the size of the phone screen isn’t large enough for distance
information to affect the rendered output significantly.

