
Apple enters the augmented reality fray with ARKit for iOS - salimmadjd
https://techcrunch.com/2017/06/05/apple-enters-the-augmented-reality-fray-with-arkit-for-ios/
======
synaesthesisx
They're paving the way for their revolutionary consumer AR eyewear (which may
be delayed now apparently). Although maybe we'll see tech from some of their
recent AR acquisitions (PrimeSense/Metaio) integrated into the upcoming
iPhone, possibly in the form of an advanced depth/surface mapping sensor of
sorts.

What's fascinating is Apple has a large team focusing on HCI & AR - what
"pinch-to-zoom" and multitouch did for smartphones is what Apple will need to
successfully pull off the "big bang" for consumer AR.

~~~
scarlac
> which may be delayed now apparently

I'm curious, what have you read/seen that makes you think this unknown product
is delayed before being announced?

~~~
threeseed
There was an AMA on Reddit from a Foxconn employee who looks to be pretty spot
on about other products:

[https://www.reddit.com/r/apple/comments/6ezhwm/iama_foxconn_...](https://www.reddit.com/r/apple/comments/6ezhwm/iama_foxconn_insider_with_information_on_next_12/)

Rough bill of materials of Project Mirrorshades - Apple Iris. Note Iris is
Siri backwards. Kopin NED Acetate frame Polarisized or prescription lens with
Zeiss smart optics Bone conduction modules Microphones (noise cancellation)
Light sensor Accelerometer for step tracking and head movement app navigation
Magnetometer for navigation Capacitive Pavel Ceramic battery Apple chipset
Charging circuit BL5 Induction module

Crystal, champagne and black. We only saw transparent reader lens but I
understand they can procure polarised or prescription lenses Cellulose acetate
Injection mold frames into an aluminum mold. Colors are added and tumbled for
finish. P3 frame design

Two sizes, men's and woman's

Usually 4x the BOM cost; it would indicate around $600 mark. Crystal is just
clear. But clear without artefacts is difficult.

~~~
sprafa
I've heard about this being real from someone who works in a very very large
corp. They are working with Apple building the first apps for the device. He
said it was not much better than Hololens at the moment.

------
roymurdock
Studied the market for AR development solutions recently. A few thoughts:

\- Probably using a lot of Metaio tech, a leading AR dev solutions provider
Apple acquired/shuttered in May 2015

\- Unfortunate naming. Sounds extremely similar to ARToolkit [1], the leading
open source/free AR dev solution.

\- Markerless is the future of AR, so good on Apple for getting this
markerless tech out to compete with Google Tango and Microsoft Hololens. The
status quo is currently marker-based recognition using "tags" or QR codes.
It's also much harder than marker-based, or "on body" (Snapchat, Facebook AR
Studio) recognition. I'd say Google Tango has some of the most impressive SDKs
out there for tracking, area learning, and depth perception, but it does
require specialized smartphone hardware from Lenovo or Asus, which
significantly limits its utility and mkt penetration at this point. Google was
just too early with Glass and got burned, but it needs a stronger hardware
platform, which is something that Apple can deliver.

\- The Unreal-engine AR video demo was cool and the graphics seemed decent for
real-time rendering on an iPad, but the real future of AR applications (read:
$$$) will come from industrial applications on wearable devices or head
mounted displays such as the Hololens, Vuzix, Daqri, or Epson Moverio devices.
Examples of industrial/enterprise AR applications: Remote help, complex
assembly, pick & pack, line monitoring, materials handling, systems training,
etc. This is the market Microsoft and PTC (Vuforia, ThingWorx) are targeting,
not the consumer gaming/advertising markets, which use more "basic" forms of
AR. This is not to say Pokemon Go/Snapchat do not generate a lot of revenue,
but it's very debatable if they can be considered true AR applications.

\- Other vendors in the AR development solutions space include: Aurasma (HPE),
Blippar, Catchoom, EON Reality, Kudan, Pikkart, Wikitude, among others.

\- In conclusion, Apple is currently stuck between a rock and a hard place.
It's somewhat late to market with ARKit, will have to compete with Facebook,
Google, and Snapchat for consumer AR-oriented developer mindshare, and will
have no play at the industrial/enterprise market if it limits its tools to
development for the iOS family.

\- The best way for Apple to alleviate this jam is to release its own AR
glasses/headset, which it is widely rumored to be developing.

[1] [https://artoolkit.org/](https://artoolkit.org/)

~~~
mendeza
For AR, to win, you do not have to be the first one to market to win, its who
can make the best platform for developers will win. As an AR developer,
previous platforms like Vuforia, ARToolkit, and even Tango are either limiting
or very hard to work with.

~~~
moron4hire
What? Vuforia is extremely easy to work with. Are you sure your difficulty
issues weren't just Unity being a garbage heap of bad UI and poorly documented
processes?

~~~
mendeza
Lol, I mean building a native ios swift app with vuforia was a pain in the
butt, since they have NO documentation on their code or any tutorials. Github
link here:
[https://github.com/interactivetech/VuforiaScenekit](https://github.com/interactivetech/VuforiaScenekit)

I have been avoiding building AR for Unity for the reasons you described xD!

~~~
moron4hire
If you can get someone to walk you through it a few times, point out the major
parts, it's the easiest way I know of to get 3D apps up and running. And it's
in very high demand.

The only problem is that I very much believe in the web application model as a
means of reaching a wide audience with the fewest resources, but Unity's WebGL
export is functionally useless.

~~~
mendeza
That would of been great to have mentorship, but I ended up doing it the
harder way, Struggling and reading code like a book. I learned a ton though,
and its taken a while.I hope to soon provide people with tutorial/walkthrough
with 3D apps.

You should take a look at AR.js, its a cool project that will hopefully
provide that wider accessibility. The challenge with Apple's ARKit is that its
AR technology( is much more challenging than Vuforia, and tricker to
implement.

------
runesoerensen
More information and docs here
[https://developer.apple.com/arkit/](https://developer.apple.com/arkit/)

~~~
stesch
.

~~~
craigching
I thought 6S+ ran A9?

~~~
bilal4hmed
It does

------
paul9290
I thought we might be seeing an AR app built into the Camera or Map app.

Nope just a kit for developers to build games only or does the ARkit allow
access to the camera app? I'm dying to see innovation from Apple like in the
camera app when you open it there's an option to view how things look in the
past(100 years ago what did this spot in New York City look like.. where you
can take a pic of yourself in the past).

I'm sure there's oodles that can be done with an AR Camera.

Apple and the others to me seem to be slow to innovate.

~~~
k-mcgrady
I might be wrong but it sounds like 3rd party devs can build those things
using ARKit.

~~~
aylmao
Yeah, and that's the whole point of it. There's too many applications of AR
for Apple to build alone.

~~~
paul9290
Hmmm but their actual camera app?

What will happen is a developer will build the app I mentioned above (could
exist already) and then Apple will later build it into the camera app. I've
never seen an app maker build functionality into any of the iPhone built in
apps.

~~~
gbanfalvi
I don't think Apple offers support to extend their camera app (there is
support for extending the photos app, however).

What you _can_ do, (and it's quite easy), is to create your own app, capturing
video from the camera. Then you can overlay any AR content you wish onto it.

~~~
paul9290
Yeah that's what I was saying.

------
htormey
ARKit was one of the most interesting features announced this WWDC. Can't wait
to play around with some demo code and see what it's like.

------
huangc10
As a developer, this was the most exciting thing for me from today's WWDC
opening speech. Can't wait to try this out myself. The demo they did with the
AR game was really neat.

~~~
satysin
It seemed quite mature for a v1 release. Then again it is hard to judge from
an Apple demo but it did look nice.

~~~
huangc10
I think there must have always been some way to do AR (Pokemon GO and other
games) but I believe it always required a team to accomplish this and possibly
work with engineers at Apple.

As an independent developer, I hope the AR kit will make it much easier for me
to create something fun and possibly integrate with Siri and other kits. This
is what I'm hoping for :D

I know VR is hot right now, but I've always had the thought AR > VR.

~~~
nerfhammer
Wasn't Pokemon GO more like pseudo-AR? Like it just superimposed an animation
with an alpha channel shadow but which otherwise had nothing to do with
whatever the camera was looking at?

~~~
MBCook
Yes. It's AR in that they overlaid something over the real world.

That's why the little bit during the demo was interesting. They showed Pokémon
go using AR but it was actually doing it correctly, where the Pokémon stayed
in position as they moved the camera around.

I was wondering what Pokémon Go would do on android. Will this feature be iOS
only or will they implement their own version on android?

------
quantumwannabe
I find it funny that Apple is just now releasing an AR SDK as when the iPhone
first came out there were a lot of AR apps available. Most of them were pretty
boring, just games like shooting spaceships or POI explorers, but they didn't
look that much more primitive than the apps demoed today. I wonder why they
didn't jump on AR back then.

~~~
MBCook
I remember playing with those on my 3G or my 4. It was pretty impressive what
people were able to do with those little phones.

I'm sure part of it is just the ability to do useful things, part of it may
simply be that they have such had room on the chips that they can do it
without draining battery life horribly. I remember those old apps clearly used
a lot of processing power.

------
tuyguntn
How does it compare to Google Tango? Tango needs special hardware, but ARKit
is not. Even if it required, there would be a definitely device (iPhone or
iPad). Traction wise Apple could win, but what about performance and accuracy
wise?

~~~
asadlionpk
For now it's more like how Snapchat's latest world filters work (try them
out). Basic environment tracking, good enough for a smooth experience,
considering you keep your camera steady.

Tango takes the tracking to the next level. Maybe future iPhones will have
that kind of IMU and other sensors, allowing Apple to expand their AR API.

~~~
Eridrus
I was pretty impressed with their live demo.

I tried Pokemon Go briefly, and their "AR" was a complete joke compared to
this.

------
agumonkey
Barely related but what happened to HealthKit ? are ambitious Apple Kits
failing ?

~~~
dpkonofa
HealthKit is still going. They announced some big wins with HealthKit on the
Apple Watch but they weren't included in the keynote. More than likely, there
wasn't enough time considering the keynote without any HealthKit announcements
was already longer than normal.

~~~
agumonkey
Thanks. Last month some medtech guy reported about different efforts and said
HealthKit vanished. The fact Apple didn't mention made me think he was right.
Now I'll wait for more.

------
enos_feedler
Does anyone have resources for "State of the art" algorithms and research that
is used to do the sensor fusion (camera frames + motion sensor data)? I found
this but not sure its the most up to date. I'm also looking for more detail in
terms of code:

[http://user.it.uu.se/~thosc112/pubpdf/holsgs2006.pdf](http://user.it.uu.se/~thosc112/pubpdf/holsgs2006.pdf)

~~~
steinomri
It seems that ARkit is actually using SLAM as the core technology (in parts,
the same technology is being used in autonomous cars and navigating drones).
Getting it to work on mobile phones is quite challenging and had been the
subject of much research in recent years, some good starting points (with
code): ORB-SLAM -
[https://github.com/raulmur/ORB_SLAM2](https://github.com/raulmur/ORB_SLAM2),
[https://www.youtube.com/watch?v=T-9PYCKhDLM](https://www.youtube.com/watch?v=T-9PYCKhDLM)
DSO - [https://github.com/JakobEngel/dso](https://github.com/JakobEngel/dso),
[https://www.youtube.com/watch?v=C6-xwSOOdqQ](https://www.youtube.com/watch?v=C6-xwSOOdqQ)
VINS - [https://github.com/HKUST-Aerial-Robotics/VINS-
Mobile](https://github.com/HKUST-Aerial-Robotics/VINS-Mobile),
[https://www.youtube.com/watch?v=0mTXnIfFisI&feature=youtu.be](https://www.youtube.com/watch?v=0mTXnIfFisI&feature=youtu.be)

(the last one being released just a few days ago...)

Of course there are a lot more resources out there and a lot of the cutting
edge research is involved with deep learning techniques (starting with e.g.
[https://github.com/alexgkendall/caffe-
posenet](https://github.com/alexgkendall/caffe-posenet)).

You may also want to watch out for this conference
[http://cvpr2017.thecvf.com/](http://cvpr2017.thecvf.com/) scheduled for the
end of this month and is usually a good source of cutting edge research...

------
Stanleyc23
There was also a very brief mention of a "depth api" before they started
talking about the actual ARKit. is that endpoint documented anywhere?

~~~
huangc10
here?
[https://developer.apple.com/documentation/arkit](https://developer.apple.com/documentation/arkit)

~~~
Stanleyc23
nah I think its something for normal photos, not immersive content.

> Camera: Updated video and image compression, improved Portrait Mode with
> support for low light, plus a new Depth API for developers.

from [https://venturebeat.com/2017/06/05/apple-announces-
ios-11/](https://venturebeat.com/2017/06/05/apple-announces-ios-11/)

~~~
photojosh
I had a look at the documentation at [1], which says "The
AVCaptureDepthDataOutput class captures and delivers depth data in a stream
(similar to how the AVCaptureVideoDataOutput delivers video data)."

So you can now access a stream of depth data. I'd put money on ARToolkit using
this stream when it's available. For now, it would only be from the iPhone 7+.

[1]
[https://developer.apple.com/documentation/avfoundation/avdep...](https://developer.apple.com/documentation/avfoundation/avdepthdata)

~~~
MBCook
AR kit isn't limited to the 7+ ( they demoed it on an iPad) space so clearly
that's not REQUIRED. I wonder if they're using the second camera for depth on
the 7+ when it's available.

Perhaps that data isn't granular enough (or fast enough) to be useful when
doing AR compared to the normal camera picture plus the motion sensor data.

------
aSig
Is anyone able to find the ARKit sample code that was mentioned during the
keynote?

~~~
rfinman
You can download the beta iOS 11 and xCode 9 and test it out. It is pretty
impressive

------
izacus
There's an AR fray? Google threw their weight behind VR on mobile, who else is
in the AR space?

~~~
aylmao
Microsoft, Snapchat, Google, Facebook.

Facebook made a big deal in F8 out of adding cameras to all its applications.
They're in the AR game big time.

[https://www.nytimes.com/2017/04/18/technology/mark-
zuckerber...](https://www.nytimes.com/2017/04/18/technology/mark-zuckerberg-
sees-augmented-reality-ecosystem-in-facebook.html?_r=0)

~~~
DonHopkins
Magic Leap is conspicuously but rightly missing from your list, because
they're a fraud that's nowhere near releasing a real product, and whose
bullshit is besmirching and dragging down the reputation of AR.

[https://www.youtube.com/watch?v=w8J5BWL8oJY](https://www.youtube.com/watch?v=w8J5BWL8oJY)

