
The iPhone X’s notch is basically a Kinect - Tomte
https://www.theverge.com/circuitbreaker/2017/9/17/16315510/iphone-x-notch-kinect-apple-primesense-microsoft
======
supernumerary
Fun story... In December 2012 I bought a Prime Sense Carmine, The Kinect for
near-distance objects like a face ... mostly to play around with Faceshift
([http://faceshift.com/studio/2015.2/introduction.html#introdu...](http://faceshift.com/studio/2015.2/introduction.html#introduction))
and
([http://www.cs.ubc.ca/~chyma/publications/ur/2015_ur_paper.pd...](http://www.cs.ubc.ca/~chyma/publications/ur/2015_ur_paper.pdf))

Apple announced its purchase shortly thereafter and the Carmine got yanked
from public sale ... thereafter Carmines were selling at a premium on Ebay, I
suppose for competitors to reverse engineer.

Been waiting for this to crop up in an Apple product...

Incidentally it is a shame an equivalent device is not available to hack on...

~~~
comboy
Bought by Apple for $360M in 2013. I've been wondering when did they start
thinking about this feature (or rather this kind of implementation for it). It
seems that even when you have a lot of money and experience, there's still a
pretty long way from an idea to its successful execution.

~~~
JustSomeNobody
Money has little to do with it. You can only add so many engineers to a
project. Therefore, some things just take time.

~~~
matude
Aka "9 women won't give birth to a child in 1 month". An example often used to
describe how adding more developers doesn't necessarily solve an issue
quicker.

~~~
gumby
Credit for this goes to Fred Brooks, and appears in his famous book "The
Mythical Man-Month"

~~~
JustSomeNobody
Yes it does. I could not find my copy to find the quote I wanted to use so I
used my own (not quite as effective) words.

~~~
gumby
Didn't mean to imply you'd done something wrong, just added to your comment.

------
IBM
There's probably a handful of acquisitions in this release cycle:

WiFiSLAM in 2013

PrimeSense in 2013

LinX in 2015

Metaio in 2015

Faceshift in 2015

Emotient in 2016

Flyby Media in 2016

RealFace in 2017

and probably more that isn't obvious to me.

It was reported by Bloomberg, funnily enough in an article framed as Apple
struggling in M&A [1], that Metaio took a lowball offer:

>Apple often refuses to work with investment bankers appointed by the seller,
preferring to deal directly with company management, according to people who
have been involved in such negotiations. Apple also dictates terms and tells
targets to take it or leave it, betting that the promise of product
development support later and the chance of appearing in future iPhones are
alluring enough, the people said.

>That was the case when Apple acquired Metaio GmbH in 2015. Bankers appointed
by the augmented-reality firm to negotiate weren’t allowed in the room, and
while Metaio executives felt the offer was low, Apple’s vision for the
technology convinced them to sell, according to a person familiar with the
discussions.

>Apple’s current M&A strategy works well for acquiring startups developing new
technology that can be added to existing Apple products. It bought 15 to 20
companies per year over the last four years. But buying larger companies
presents a different challenge, particularly if there are rival bids. Bankers
often diffuse tension between bidders and targets, but Apple’s approach can
make that process difficult.

>“There’s a swagger -- you may call it arrogance -- about the culture there,”
said Risley of Architect Partners. “They’re used to being able to muscle their
way in and get attractive economics.”

Which seems completely logical on Metaio's part. It's obvious a lot of these
startups working on fundamental technologies are just going to toil in
obscurity, and selling to Apple is a chance to have your technology deployed
and used in the biggest way possible.

[1] [https://www.bloomberg.com/news/articles/2017-02-15/apple-
str...](https://www.bloomberg.com/news/articles/2017-02-15/apple-struggles-to-
make-big-deals-hampering-strategy-shifts)

~~~
Steko
It hasn't been confirmed but it's highly likely the X has a liquidmetal back.
Apple's had an exclusive arrangement for quite awhile and this would be the
first significant use of it.

~~~
TheCoreh
Isn't the back glass to allow for wireless charging?

~~~
Steko
Primarily yes but, to be clear, liquidmetal is a (metallic) glass.

So yes Apple's main reason is for wireless charging but they also want to
differentiate where they can and liquidmetal gives them "the hardest glass in
a smartphone ever." (Ive quote).

Analysis and background including a very relevent patent application made a
year ago and published in March for "using Liquid Metal (Metallic Glass) for
the Backside of an iPhone":

[http://www.patentlyapple.com/patently-
apple/2017/09/apples-l...](http://www.patentlyapple.com/patently-
apple/2017/09/apples-liquid-metal-backside-flexible-wraparound-oled-display-
for-reducing-the-bezel-are-patents-fulfilled.html)

------
fencepost
This is the reason that even the makeup, glasses, etc. that people have come
up with to counter facial recognition are going to be completely ineffective
in a few years.

Contemplate if you will a hallway immediately past Customs in an airport.
Equip it with multiple sensors of this type to provide complete coverage and
redundant imaging/sensing. While you're at it, set up gait recognition as
well. Then correlate the received profiles with the passport/identity data of
the people just passed through customs.

Congratulations, you've just started building a database of everyone entering
the country with biometric data that can be checked in the field with
equipment costing less than $1000, and which can later be cross checked to
find people traveling with false or multiple IDs ("this facial structure comes
back as matching (person x) and (person y), and the gait is almost the same.
Pick him up."), and it could all be done with technology that exists today.

Edit: autocorrect

~~~
blitmap
I wish as technology becomes available which could be used for more Orwellian
surveillance someone stood back and thought about ways it could be used for
good rather than defending some absurd fear of terrorist attacks.

Like... I remember a few years back there was an article about using high
framerate cameras to detect the heartbeat of people the camera viewed in
hospitals. I wish surveillance could be used to watch people at risk for heart
attacks, bring it to their notice if one is detected in the early stages, and
direct them to their nearest hospital - in a way that involves a concert of
technologies/devices.

Instead of CCTV cameras on every corner being used for criminals, use them to
watch for health risks? Alert the person through their Apple Watch, send
directions to the watch and phone, alert nearby first responders, notify
family, etc... A route could even be established for an ambulance ahead-of-
time, rather than as an ambulance rolls up to a signal.

I'm tired of worrying about terrorists. Maybe I want to be blind to the small
chance a carbomb is possible. I know the most innovative stuff is often a
result of defense spending/planning, but I hate thinking a driving force
behind technology today is people afraid of other people.

</end-rant>

~~~
Tsiklon
Unfortunately the cynic in me says there's more money in keeping people
fearful.

~~~
iammyIP
Nah, i don't think that's it. Fear is a feeling, which is per default not any
more or less connected to "i need to buy this thing" than any other feeling.
So why should fear encourage spending money more than any other feeling. How
about the idea that a joyful person is much more inclined to spend money - but
on different things. Just imagine a world without any fear, just with
different kind of markets and the same amount of money.

~~~
astrobe_
> Nah, i don't think that's it. Fear is a feeling, which is per default not
> any more or less connected to "i need to buy this thing" than any other
> feeling

It seems to me that a significant part of the press and TV news are selling
just that - fear. It doesn't take long to find a word like "threat", example
taken at random, on the first page of the NYT ("Shinzo Abe: Solidarity Against
the North Korean Threat").

~~~
chii
If you have fear, then you'd be more willing to give up certain freedoms to
rid said fear. Freedoms such as privacy. Mark my words, there's going to
definitely be massive facial recognition on the population, in the name of
keeping you "safe".

------
vmarsy
Not to be alarmist, but how do we know this IR blasting won't be damaging to
the eyes, a 3 seconds google search leads to a 2011 study[1] which concludes:

> The protein of eye lens is very sensitive to IR radiation which is hazardous
> and may lead to cataract.

[1]
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3116568/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3116568/)

~~~
joshvm
In short because an optical engineer at apple checked that the emission was at
a safe level, and if it wasn't they'd be shafted by lawsuits. Assuming it's a
laser, it'll be a Class 1. The emitter is short range, so the power doesn't
need to be as high (though interesting to see how well it works outdoors).

> Workers in hot environments, exposed to IR, developed lenticular opacities
> due to IR irradiance in the order of 80–400 mW/cm2 on a daily basis for
> 10–15 years

That is _really_ high. The original Kinect has an output power (at the
emitter) of around 60mW. The expanded beam is safe to look at beyond a
centimeter or two (I think less, actually) due to energy conservation.

On top of that you're only being exposed for a second or two to grab the depth
image, not 5 minutes.

~~~
kabdib
The Kinect has a number of hardware interlocks as well, and will shut the
laser down (without firmware intervention, because firmware might not be
working) based on some hard-wired detections. The unexpanded raw laser in the
Kinect, prior to hitting the hologram, wouldn't be great to look at.

~~~
joshvm
Sure, in my post I assume it's going through the diffractive optical element
(DOE). A 60mW, 940nm laser is absolutely not class 1!

------
wyc
I was an early backer of the LiDAR Lite[0], and I'm really excited to see
LiDAR products become more affordable. A recent project called Sweep[1] the
$350 super low-end disruptor to the $80,000 Velodyne models. I wonder how long
before we have plug-and-play open-source projects for multi-sensor fusion[2]
of cameras, LiDARs, and microphone arrays. A common digital trend appears to
be subsidizing sensor quality with data volume and processing power. Are there
projects for this now?

[0]
[https://www.sparkfun.com/products/14032](https://www.sparkfun.com/products/14032)

[1] [http://scanse.io/](http://scanse.io/)

[2]
[https://en.wikipedia.org/wiki/Joint_Probabilistic_Data_Assoc...](https://en.wikipedia.org/wiki/Joint_Probabilistic_Data_Association_Filter)

~~~
Judgmentality
The sensor in the iPhone X isn't LiDAR, it's structured light.

[https://en.wikipedia.org/wiki/Structured-
light_3D_scanner](https://en.wikipedia.org/wiki/Structured-light_3D_scanner)

~~~
wyc
Thanks, and I can't wait to see systems that will fuse cheap multi-pose
structured light sensor data together too! These devices are all measuring
proxies of the same thing: 3D structure.

~~~
zackya89
Does Tesla still use Lidar ?

~~~
detaro
Have they started? I thought they insisted that cameras are good enough?

~~~
awalton
Cameras are definitely not good enough alone. Tesla's enhanced their camera
systems with different models of automotive-range radar (most likely
manufactured by Bosch; either the LRR4 or the MRR1).

------
siscia
Isn't this tech more useful on the other side of the phone?

It is true that you may try to catch emotion from the user face but on the
other side you can catch the world...

~~~
cududa
Use to work on the Kinect so have some knowledge around it - not really. The
lenses and power requirements to do more than a couple feet away would be
huge. Though I'm sure they'll eventually get it.

~~~
cma
From cursory reading, laser illuminators can be ~50X more efficient than the
IR LEDs in Kinectv2 (though v1, ehich they compare it to, used IR lasers too,
and didn't need active cooling like v2, it can also spread the illumination
over a longer exposure than is required for v2's time-of-flight) and
illuminate a much more narrow band which allows you to use a more selective
filter, so you have less background IR to overcome.

It still may be only good to a few feet out in direct sunlight, I don't know.

------
anfractuosity
I think the new Kinect makes use of time-of-flight if I recall correctly
(which they do indicate in the article). If I understand correctly, with time-
of-flight you don't need to project a pattern as in structured light setups.

~~~
Animats
Yes, the current Kinect is a time-of-flight device. The original one was an
"unstructured light" device, the generic term for those random-pattern-of-dots
projectors. This is a takeoff on "structured light", where you project a known
line pattern on a 3D surface and view it from another angle to get depth.
Structured light systems are used industrially; the compute power required is
low, as is the software complexity.[1]

[1] [http://www.micro-epsilon.com/2D_3D/laser-scanner/](http://www.micro-
epsilon.com/2D_3D/laser-scanner/)

------
intended
Wait - so i could just "blast" a suitably depth adjusted copy of a face scan
on a white page and get the iphone to unlock?

Basically use a surreptitious infrared camera, copy the face of a person, then
project it onto a screen and I would mostly be good to go?

Alternatively, if I just plastered infrared dots on someones face, face
recognition would cease to work? So 2 Iphone X could be used to interfere with
each other?

~~~
IshKebab
"Just"

~~~
intended
Fine. Just is hyperbole. But I suppose I could just interfere with logging in
if I had 2 iPhone X and pointed both at a persons face.

I'm guessing The 2 sets of dots would overlap and confuse.

~~~
IshKebab
Actually probably not. At least not fatally. You can apparently use two
Kinects simultaneously with no serious issues.

------
joshumax
I've always thought that depth sensing tech could be used to help offload some
of the nasty hacks floating around in the world of AR, computer vision, and
spatial navigation. I just never realized _how much_ it would speed up
development until I decided to play around with an Orbbec Astra and PCL a few
years ago... I'm not an Apple user, but I am glad that they implemented this
technology in the iPhone X since it will spur other vendors to adopt this type
of technology as well. Hopefully they will allow some form of direct access to
the IR dot projector and camera along with DepthKit or whatever Apple decides
to call it. Until that time, however, at least other projects like
structure.io are attempting to bring depth sensing technology into the
mainstream.

~~~
chicago_wade
> Hopefully they will allow some form of direct access to the IR dot projector
> and camera along with DepthKit or whatever Apple decides to call it

And then the app developers upload the depth data to their servers and use it
to track users, and then the servers are hacked and the depth data is taken by
the hackers and then the hackers sell the depth data and then someone can use
that data to unlock stolen iPhones. Sounds great /s

------
shaimagz
It's 1/100 of the Kinect's size, that is the innovation and a pretty big leap

~~~
awalton
Meanwhile everyone forgets Project Tango which has a similar sized sensor and
structured light projector...

The Apple "innovation" here is finding an application for it: basically
Snapchat. (Because, and let's be really serious here, FaceID is not a good
application.)

...and that alone just shows how off the mark this implementation is. When
this comes to the cheap Chinese Android phones in six months, it's going to
cause another sales boom there, as parents will actually be willing to spend a
few hundred dollars for a phone upgrade for a teenager, verses the thousand
bucks for one of these Apple phones...

~~~
jonknee
> FaceID is not a good application

Why not? More secure face recognition seems like a very good application that
will be useful to millions.

~~~
1_2__4
Can you explain how?

~~~
mcintyre1994
iPhone X doesn't have touchID, if you want security and don't want to enter a
password/pin then you have no choice.

~~~
seba_dos1
Using biometrics for security is hardly secure anyway.

------
Asdfbla
Were there no patent problems? I'm glad about it, of course, but it just
somewhat surprised me that so many companies can use the same basic principle
and there didn't seem to be too much hassle with people suing each other. Then
again, I might just not have heard of any high-profile cases.

The basic working principle is probably also basic enough to have lots of
prior art I guess.

~~~
jccalhoun
In another thread I wondered if apple had to license any tech from microsoft
but got voted down.

~~~
djrogers
Apple bought PrimeSense- the company who’s tech MS Alice SEs for Kinect. So
no.

------
sschueller
Wasn't the kinect susceptible to infrared interference such as from sun light?

Question is, will there be issues with face unlock in bright sunlight.

~~~
mcphage
I guess it's possible that Apple never considered the Sun when they were
designing this.

~~~
rmrfrmrf
Blasted edge cases!

~~~
sytelus
Another one: If you see someone's phone lying around, just stare at it few
times and see if yo disabled their FaceID without ever touching it :)

------
partiallypro
I wonder if they licensed any of this from Microsoft. I know they share a
cross patent agreement on various things; but they bought PrimeSense, however
that doesn't mean nothing was licensed.

~~~
rasz
Other way around. Microsoft screwed project Natal badly, spending >$1B to buy
2-3 companies in a row and still unable to make Kinect. In the end they were
forced to license from Primesense, because alternative was no Kinect at all.

------
fijal
Can you program it like a kine ct? Or is it essentially a kinect running one
program, which makes it a lot less like a kinect (genuine question, I haven't
been able to find any info)

~~~
bdibs
You can play with it using ARKit [0].

[0]
[https://developer.apple.com/documentation/arkit](https://developer.apple.com/documentation/arkit)

~~~
dhritzkiv
However, only a depth map is accessible by developers. The raw sensor data is
not.

------
ryanmarsh
When the Kinect first came out I immediately thought to myself "one day all
the tech in this thing will be cheap and tiny, and that day will be a weird
day". I know it's obvious, tech gets smaller and cheaper. No great insight
there. I just think it's one of those small innovations that ends up having an
outsized impact.

Think of it this way, what might be possible or normal once sensors like this
are so cheap they come embedded in all OTS camera modules for even the
cheapest of devices?

Pretty crazy right?

------
tenryuu
Interestingly enough, I made a similar comparison before reading this article
on an IRC. It felt similar to Windows Hello which uses an RGB, IR and 3D
camera for face detection, which are all used for authentication. So I just
made a quick bastardisation of the Kinect on this camera set, then applied
that to FaceID as well

------
skeletonjelly
Like others have mentioned the Kinect v2 uses time of flight to detect depth.
I'd be more impressed if Apple used this over replicating v1 of the Kinect.

Even though I'm an Android user, I think I'm more scared about Google catching
up and incentivising the high res scanning of faces for some consumer
application.

------
rch
> _Google 's Tango technology ... which is also based on infrared depth
> detection_

I thought Tango included hardware to process parallax and phone position so it
could potentially work outside and in bright open areas.

~~~
gggdvnkhmbgjvbn
I own a tango phone and it sucks outdoors. Last time I tried was a few months
ago though

------
Abishek_Muthian
Of course it is, apart from hardware comparisions; Apple has all the necessary
components sans gamepad to create an Nintendo Switch esque setup.

------
macromaniac
Depth sensor is cool, but I think a better feature would be increasing the EM
spectrum the phone could operate on. It would be nice to open garage doors,
detect speed traps, or enable push to start on cars.

~~~
lisnake
Steal car alarm codes with an app...

~~~
macromaniac
Eh I was mostly sick of having to have a device to open the gate, a device to
turn on my car, and a device to open my garage. Whats wrong with 300MHz? I'd
rather not have to carry around all this crap vs a depth sensor that I might
use once as a gimmick to 3d map my bathtub.

~~~
rizwank
Radios and antennas. More weight and power for limited usecases. In Apple and
Google’s mind - replace your garage door opener with a connected one. You are
talking about an SDR to allow multiple frequency ranges.

~~~
TeMPOraL
I personally would love to have an SDR integrated with my phone (and for
operating in lower frequency ranges, just give me an antenna port next to the
audio jack (oh, wait...)). But then again, a TX-enabled SDR in a popular
consumer device? FCC would not be happy...

------
m3kw9
Easy for them to say such things and devalue the engineering effort that goes
into miniaturizing such tech and having the software to match it.

~~~
gnodar
Interesting. The way I read the article, I got the sense that it was
emphasizing exactly that, by highlighting examples of how far the tech has
progressed in less than 10 years.

------
senatorobama
Does anyone collate a list of startups (in newfangled tech) likely to be
acquired by Apple? Would be good to work for a place where a nice earn-out is
likely.

~~~
amrrs
I'm sure M&A analysts in Consulting Companies would be collating this to
enable their next pitch!

~~~
senatorobama
Wait, so THAT's what M&A do? I thought they simply facilitate transactions
once management has decided who to acquire. Which firm does M&A for Apple?
That would be a great job.

~~~
amrrs
"We help clients ensure that their M&A strategy aligns with their broader
corporate strategy. We identify and assess targets based on a client's
strategic objectives, potential synergies, organizational and cultural fit,
and the feasibility of a deal. To help the transaction proceed smoothly, we
support clients in structuring the deal, communicating its rationale to
stakeholders and markets, and planning for integration."

\- Mckinsey's website [http://www.mckinsey.com/business-functions/strategy-
and-corp...](http://www.mckinsey.com/business-functions/strategy-and-
corporate-finance/how-we-help-clients/transactions)

~~~
senatorobama
Sounds like a pretty good job.

------
thinbeige
Wondering if Apple should have rather spent their time and energy in R&D for a
front-facing camera sitting behind an OLED screen so we could have a real
bezel-less screen.

Earspeaker behind a screen works already (a Chinese handset manufacturer did
this) and I dont need the other sensors and that for Face-ID. I found a
fingerprint sensor better anyway.

~~~
shanev
Apple has a patent on a camera integrated into the display. So it’ll happen at
some point. There’s probably a team working on it in R&D as we speak.

How did you find TouchID to be better than FaceID? Did you get a pre-release
version of iPhone X?

