
Show HN: A tiny JS library for real-time localization of eye pupils - tehnokv
https://tehnokv.com/posts/puploc-with-trees/demo/
======
lioeters
For an explanation, see author's post:

Using decision trees for real-time localization of eye pupils

[https://tehnokv.com/posts/puploc-with-
trees/](https://tehnokv.com/posts/puploc-with-trees/)

It's actually quite fascinating. I'm too paranoid to enable webcam for any
random site, so will be trying it out locally.

For face detection and determining the locations of the eyes, the demo uses
picojs, described in another post:

pico.js: a face-detection library in 200 lines of JavaScript

[https://tehnokv.com/posts/picojs-intro/](https://tehnokv.com/posts/picojs-
intro/)

~~~
errantspark
I'm all for being paranoid but what exactly is the threat model here? Someone
is going to have a few seconds of video of you?

~~~
taneq
Spearfishing family members using a pic and/or some audio of you to generate a
fake "help me" video? (I mean, that probably won't happen, just trying to
think of something an attacker could do with it.)

~~~
dymk
Imagine being paranoid to the point of actually entertaining a hypothetical
like that, so much that it affects your browsing habits.

~~~
drawnwren
There is actually already a pretty prevalent (lower tech) version of this
scam. Grandparents are called by a con artist who attempts to pretend to be a
(previously unidentified) grandchild in distress in a foreign country. [1] My
grandmother lost a few thousand dollars to it because she thought they were
me.

1 -
[https://www.ag.state.mn.us/Consumer/Publications/GrandParent...](https://www.ag.state.mn.us/Consumer/Publications/GrandParentScams.asp)

------
mlang23
The next tracking technology that completely fails on me. Tracking mouse
movements is already totally helpless when it comes to blind users. But
tracking my eye pupils is even more useless. I just hope nobody will ever
impelment CAPTCHA based on this. It is already hard enough (and sometimes
impossible) to proove that I am human.

~~~
erklik
I was thinking of using something like this that would track my pupils and
then switch window focus i.e. allowing me to switch from terminal to terminal
without doing Alt-Tab.

Yes, probably the height of laziness to avoid pressing Alt-Tab but its
interesting to me.

Edit: Its possible that i am mistaken and what I would need for that is more
gaze-tracking than pupil-tracking.

~~~
e12e
The interface in vr for elite:dangerous works like this, it does gaze tracking
of sorts (tracking the headset) and pops up context menues (weapon systems,
radio/communication, navigation etc). It works really well imnho. Otoh the
interface is similar in regular 2d, but feels clunky and gimmicky there.

~~~
erklik
That sounds interesting. I will take a look at it. Honestly, I haven't done
any study or reading on this topic so have essentially zero idea of how it
might work, especially when using a webcam.

------
umvi
Can't wait to integrate this with ad serving tech to make sure users are
actually watching the ads. Imagine adding little "eye catchers" to ads, and if
the pupils don't respond, they are likely looking at a different screen or
looking away from their computer which means you can safely pause the ad until
you have their attention again! The possibilities are endless - integrate with
mobile games and reward users whose pupils follow the eye catchers with more
gems and more ads!

~~~
stordoff
Microsoft had a patent application on a very similar idea:

> Television viewing tends to be a passive experience for a viewer[...] To
> increase interactive viewing and encourage a user to watch one or more
> particular items of video content, awards and achievements may be tied to
> those items of video content. [...] Producers, distributors, and advertisers
> of the video content may set viewing goals and award a viewer who has
> reached the goals.[...]

> Additionally, the viewing behavior may include an action performable by the
> viewer and detectable by one or more sensors, such as a depth camera.[...]

> If the viewing goal has been met, an award may be granted to the viewer. An
> award may be a virtual award. Such as an addition to a viewer score or an
> update to an avatar associated with the viewer

[https://patents.google.com/patent/US20130125161](https://patents.google.com/patent/US20130125161)

------
rodw
I'm always curious about what it takes to confuse things like this (if for no
other reason so that I will know how to make face recognition camouflage to
hide from the autonomous weapons that roam our future post-apocalyptic
hellscape).

This demo seems to do pretty well -- e.g. it handles covering one eye or
arbitrary parts of your face, but has some trouble at oblique angles (like
your face in profile or near-profile).

But here's an odd thing I noticed:

\- Take a pair of over-the-ear headphones ("cans").

\- Rotate them 90 degrees so that one earpiece is on your forehead (or above
it) and the other is at the back of your head.

This seems to completely stump the algorithm, even when your eyes are plainly
visible and looking straight into the camera. With my headphones turned
sideways it never once identified my pupils (or my face) accurately.

Possibly a hat would do something similar (I don't have one handy) but a
largish dark circle your forehead seems to confuse it.

Very neat for a 200 LoC browser-based demo.

~~~
franciscop
This seems to have a lot of trouble when wearing glasses (large-style
glasses). I was surprised of how bad it seemed for 2019 tech, then tooky
glasses away and was blown away of how good it is.

~~~
rodw
It seemed to have a little trouble with light reflecting off my glasses
(sometimes it found my eyes but placed my pupils more or less on my eyebrows)
but it generally did OK with my glasses.

My glasses are closer to the "invsible frames" side of the spectrum than the
Harry Caray / Elvis Costello style, so maybe that matters.

I guess maybe I should just review the code but I was trying to figure out if
it uses other facial features to "anchor" the eyes or just the (fairly
recognizable) features of human eyes themselves. Your glasses experience makes
me guess it is pretty eye dependent -- it does have more trouble with one eye
covered for example, whereas if it was anchoring off some combination of
ear/nose/mouth/chin it should be easy enough to ID the eye from half the face,
let alone with just an eyepatch-sized cover.

------
mncharity
Nifty. I use PoseNet[1] on tensorflow.js to locate eyes for 3D UIs on a
laptop. lploc seems much faster, but requires a more restricted head pose.
Maybe I'll try using both.

I've also been looking for usable gaze tracking, to play with combined gaze
(fast but low resolution) and head (high resolution but slow) pointing.
Perhaps lploc can help with that.

So thank you for your work.

[1] PoseNet webcam demo: [https://storage.googleapis.com/tfjs-
models/demos/posenet/cam...](https://storage.googleapis.com/tfjs-
models/demos/posenet/camera.html)

------
mbrookes
The author means real-time _location_ of pupils. This is not gaze tracking
though.

~~~
azhenley
Gaze tracking on browsers with webcams does exist though!

[https://webgazer.cs.brown.edu/](https://webgazer.cs.brown.edu/)

~~~
mncharity
I finally got webgazer to work once, a year or so ago. As I recall, it
required non-default settings, shaving my beard, adding lights pointed at my
face, hanging a backdrop, and holding eyes and bushy eyebrows just so. And
maybe something else I'm failing to remember. The joys of small training sets.

No doubt it "just works" for some people. For others, not so much.

------
tylerchilds
I'm sure you know this, but just in case:

If you add this CSS to your site, you can increase the readability without
sacrificing the aesthetic.

body { max-width: 65em; margin: 10px auto; }

------
cookingrobot
It works pretty well, but it has the same issue I see with a lot of these
object tracking algos - it jitters all over even when I’m holding as still as
I can. Why are these systems so sensitive and unstable? Could the algo be
trained to prefer similar output for similar inputs to reduce the flicker at
the cost of some accuracy?

------
haberdasher
What am I missing? "Click the button below and allow the page to access your
webcam."

...and then?

~~~
cstrat
If it works, you will see a live feed of your webcam with your face encircled
with your pupils highlighted as red dots... its pretty impressive!

I'm guessing your browser is blocking the webcam, or maybe its not configured
correctly to work with your browser. did you get a popup requesting permission
from your browser? sometimes its hidden at the edge of the URL bar behind an
icon.

------
emilfihlman
Not very good if you change head orientation or rotation but otherwise pretty
cool!

------
dlhavema
Doesn't work on *my mobile android chrome. I'll try again at work tomorrow.
Looks pretty cool though.

I would love an interaction method like this.. I can already think of some
super fun UIs to build around it...

~~~
osrec
Worked pretty well for me on my low-end 2 year old Samsung!

------
Deukhoofd
Works on Chrome, but doesn't appear to work on Firefox. I can see my webcam
feed, but the red indicators don't show up.

Edit: Nevermind, it requires access to canvas data which for me is blocked by
default.

~~~
Estyn
Works on firefox for me 67.0.4 (64-bit)

------
Lowkeyloki
Sorry if I don't trust some random web page to use my camera because they
promise they're not logging it to their server.

~~~
skeletonjelly
What if you open incognito mode, open the page, cut off your network, then
close when you're done with the demo?

~~~
ASalazarMX
I overread and I thought you were being sarcastic with "[..], then close your
eyes when you're done with the demo".

~~~
skeletonjelly
I overuse sarcasm in reality to the point that people don't believe me when
I'm sincere, so no surprises I didn't write this out clearly enough :D

Seriously though I think if it's all JS based you should be good to use
incognito and offline to gate off the code from uploading.

------
your-nanny
so, we use eye tracking in experiments to look at how people acquire
information. For example, you can parameterize a drift diffusion model using
input from gaze. But the equipment is not cheap. I see this as a means of
acquiring that sort of data on the cheap. For example on mturk.

------
mevile
This is the kind of tech that managers might like to install on their
employees computers. If you're smart enough to build interesting things, maybe
spend that talent on something less creepy.

~~~
edwintorok
Doing this in javascript is creepy, however the technology itself can have
other than nefarious uses too. For example in a bandwidth constrained
situation you could determine which piece of a video stream to optimize for
higher quality based on the general area you are or were looking at, if you
look at it long enough it should stop being fuzzy.

~~~
ASalazarMX
Video streaming is too slow to adapt, but I see a way for advertisers to check
if you watched an ad even if you didn't interact. Specially if the site
already uses the webcam.

~~~
userbinator
That was the thought that immediately came to mind when I saw this article ---
as if unskippable ads were bad enough, with this they'll be able to make ads
that you can't _not_ watch, and the whole thing just disturbs me greatly. No
doubt it'll get spun into an ostensibly "user-friendly" "automatically pauses
the video if you look away" "feature"... but the motives are clear enough.

~~~
lbotos
Cue black mirror episode:
[https://en.wikipedia.org/wiki/Fifteen_Million_Merits](https://en.wikipedia.org/wiki/Fifteen_Million_Merits)

