
Oculus: Hand Tracking Updates - karanganesan
https://developer.oculus.com/blog/hand-tracking-updates-accepting-app-submissions-starting-may-28/
======
latchkey
Oh nice! I just bought Quest over the weekend from a guy off craigslist who
was selling them at cost and still wrapped in plastic.

Fantastically impressed with things. Totally worth the $500. The games are
amazing, but what is really impressing me is all the vr video content. The
ability to "stand" on a stage while the orchestra plays Mozart all around you
is mind blowing.

It is neat to see them pushing the boundaries with updates.

~~~
Rebelgecko
The Quest is probably the first thing I've bought in a long time that has
given me that "Christmas morning" feeling. It's such an impressive piece of
tech, and VR without wires is so much more immersive, even if it limits the
graphical capabilities

~~~
alexcroox
Don’t forget if you install Virtual Desktop you can play all your PCVR games
wirelessly on it too. I play HL Alyx like this and it’s flawless.

~~~
joeskyyy
Wait really?! I was wondering why I would want to fork over so much money for
the app, but having this as a possibility changes everything. Thanks for the
heads up :D

~~~
someperson
To wirelessly stream PC virtual reality games to the Oculus Quest you wouldn't
just need Virtual Desktop but also a 5GHz wifi router.

You may be interested in the alternative (official) solution, which is not
wireless but allows streaming PC VR games using "Oculus Link" using a
compatible USBC cable. The cable that comes in the box is unfortunately USB2.0
(so not compatible), your best bet is to checkout this page:
[https://support.oculus.com/444256562873335/](https://support.oculus.com/444256562873335/).
The official Oculus cable has fibre-optic transceivers built in so is very
expensive. Oculus lists some third-party cables on Amazon by Anker, which used
by a lot of people (10 foot recommended, but the tiny 3 foot one is really
cheap).

I haven't yet tried Virtual Desktop 5GHz streaming, but Oculus Link with a 3
foot cable streams perfectly (good for seated VR, which many PC games assume).
I do want to try 5GHz Wifi streaming though! Untethered play is the best!

~~~
vlovich123
That's no longer true. The latest v17 update allows you to use USB2 (including
the in-box cable).

[https://www.reddit.com/r/OculusQuest/comments/gjgcq0/oculus_...](https://www.reddit.com/r/OculusQuest/comments/gjgcq0/oculus_link_usb_20_support_on_ptc_170/)

Disclaimer: I work at Oculus but do not represent my employer on public
forums. Opinions are my own.

~~~
someperson
Wow, that's insanely cool! Thanks for letting me know!

------
mark_l_watson
My Oculus Quest is the best toy that I have bought for myself in years, even
more enjoyable since I did VR for SAIC and Disney about 20+ years ago, so
super exciting to see such a fun low cost consumer VR device. I can’t wait to
try a game/experience using hand tracking. Lot’s of great content, my
favorites being the Darth Vader trilogy and the RX racket ball and ping pong
games.

------
gegtik
I really want to know why this is feasible for the Quest but not the Rift S,
given that both have Inside-Out tracking (ie. IR cameras mounted on the
headset). In fact, the Rift S has 5 cameras to the Quests's 4 so it should be
at least as capable shouldn't it?

~~~
shafyy
I would be suprised if it's primarily a technological reason.

My thinking is: The Rift S is 99.99% used for "serious" gaming and PCVR will
always be that way. For serious gaming you'll probably need some kind of
controllers for the near future, since you want that fidelity and haptic
feedback. Now, controllers could be replaced some day by braceletes or what
not, but this is not going to happen next year.

The Quest and its successors (i.e. stand-alone VR), on the other hand, is and
will be more and more used for lighter gaming, video, browsing, productivity,
fitness, social etc. Often times, hand-tracking will be enough without the
need for controllers.

Therefore, they focus their resources on bringing hand-tracking only to the
Quest.

~~~
zmmmmm
My guess is it's partly that but also mixed in with aligning to their mass
market consumer ambitions.

I've given my Quest to countless people to try out and while every single
person is blown away, many / most of them are challenged to use the
controllers as anything more than simple pointing devices without significant
training. With hand tracking that all goes away. You just use your hands how
you normally use them and it "just works". It's similar to how the original
iPhone captured everybody's imaginations because it offered such intuitive
gestures to control its UI.

If you see the Quest as their "mass market" device and then you consider that
their top priority is to knock over every barrier that stops it spreading like
wildfire, then it's a no brainer to eliminate the controllers.

------
ninenines
Not exactly as powerful but something similar and that can run on a browser,
opening up to a lot of creative applications, handpose powered by
tf.js(similar model exists for tflite)

Demo: [https://storage.googleapis.com/tfjs-
models/demos/handpose/in...](https://storage.googleapis.com/tfjs-
models/demos/handpose/index.html)

Repo: [https://github.com/tensorflow/tfjs-
models/blob/master/handpo...](https://github.com/tensorflow/tfjs-
models/blob/master/handpose/README.md)

Tflite model:
[https://github.com/google/mediapipe/tree/master/mediapipe/mo...](https://github.com/google/mediapipe/tree/master/mediapipe/models/hand_landmark_3d.tflite)
there is lighter 2d model as well

------
melling
With the worry about Coronavirus, hand tracking with cameras or something like
Google Soli[1] might be useful in ATM’s, elevators, kiosks, etc so people can
avoid physically touching the devices.

[1] [https://atap.google.com/soli/](https://atap.google.com/soli/)

------
rgovostes
Apple's first published ML paper[1] described how they trained a neural
network to estimate eye gaze. To augment the limited data set of labeled eye
photos, they rendered synthetic eyes and then used adversarial training to
make them match the real-life data set.

This paper came out several months before the iPhone X introduced Face ID with
eye gaze tracking, so it can be seen as a hint of things that were to come.

What's intriguing to me is that the paper also talks about doing the same with
synthetic depth maps of hands, adding details such as occlusion and
imperfectly-captured edges. With Apple investing heavily in ARKit, and rumors
of an AR headset, it seems like hand tracking is inevitable.

1: [https://arxiv.org/abs/1612.07828](https://arxiv.org/abs/1612.07828)

------
nkurz
Does the Hand Tracking API allow for tracking other objects? In particular,
could a long straight object be tracked as a pointer based on tracking two
points along its axis? This seems like it would be computationally easier than
tracking hands, and would open up a lot of possibilities for using real world
objects as props within the VR environment.

~~~
GraffitiTim
I imagine some kind of VR tracker for the Quest will come out eventually.
Something like the Vive trackers but smaller. Agree, it would open up all
kinds of interesting applications.

~~~
pharke
They could be wildly successful if instead of $100 pucks they could just sell
$1 stickers that you could attach to whatever you want. With enough of them
you could probably use it to aid scanning in geometry of real world objects

------
texascloud
Are there cameras on the sides of the occulus so peripheral hand movements can
be recognized?

~~~
krasin
Yes, pretty much:
[https://thumbor.forbes.com/thumbor/960x0/https%3A%2F%2Fblogs...](https://thumbor.forbes.com/thumbor/960x0/https%3A%2F%2Fblogs-
images.forbes.com%2Fsolrogers%2Ffiles%2F2019%2F05%2F190402_Quest_FH_dark_color-1200x675.jpg)

------
sytse
With hand tracking they mean finger tracking, seeing you 'click' with your
index finger and thumb. The Oculus has hand tracking today via controllers
already.

~~~
Impossible
They mean handtracking. Oculus touch is a tracked controller, which means the
controller is tracked and the tracking system has no concept of a hand. You
could put the controller on your foot, or the floor or on your dog and it
would track in the same way as your hand. Hand tracking uses computer vision
to recognize hands, for most humans fingers are included in that

~~~
TeMPOraL
And Quest has that proper hand tracking already (had it for something like 5
months now), it's just been an experimental feature until now. I haven't used
it much; for some reason the tracked hands seem to be placed lower than the
fake hands when I'm using controllers, which makes the latter a better
experience for me.

~~~
Impossible
Increased latency due to using complex, deep learning based, entirely computer
vision driven hand tracking vs. tracked controllers which partially use
accelerometer data to correct drift and account for latency, and use a much
faster classic computer vision algorithm for position tracking
([https://developer.oculus.com/blog/tracking-technology-
explai...](https://developer.oculus.com/blog/tracking-technology-explained-
led-matching/))

~~~
TeMPOraL
Do Deep Learning-based methods introduce all that much latency? My
understanding of neural network is that the training is expensive, but using
them is essentially free - you have to push your pixels through a few layers
of matrix multiplications and you have the results; it's just tuning those
matrices that takes time.

~~~
Impossible
VR latency is measured in milliseconds, the operation you described can easily
take multiple frames to complete even with a very fast, optimized model.
Machine learning inference is never free, and often isn't real-time. Training
can take hours to weeks depending on the model.

------
bsanr2
At the risk of being a downer: I'll be so happy once we move past the
"preschool playset" era of hand-tracked software.

I wonder what it is that's keeping us from moving past these dull tech demos.
Is the issue the interaction design being too simplistic ("Look, I can pick
[thing] up!")? Or perhaps the opposite: that, without suitable haptics, too
much energy has to be devoted to a very unnatural way of interacting
(essentially with phantoms). It seems that games like BeatSaber and apps like
TiltBrush have cracked the code, but you have to wonder how.

