
WebGazer.js: Eye Tracking on the Browser - trymas
http://webgazer.cs.brown.edu/#
======
o_____________o
The demo you came here for

[https://webgazer.cs.brown.edu/collision.html](https://webgazer.cs.brown.edu/collision.html)

Click around the screen where you're currently looking to calibrate

~~~
martin-adams
That is amazing. I've never used an eye-tracking device before so the
experience of hitting the circles with my eyes is very surreal.

------
wrigby
I think this would be really useful to UX researchers! The folks I know in
that space often set up screen-sharing sessions with users, and watch them
interact with the UI in real time. Adding gaze tracking to this process seems
like a natural progression, allowing the researchers to see where the user
expects a button or UI feature to be, even if the user can't articulate that
in real time.

------
travjones
I think this is impressive work. My lab does eye tracking with really
expensive equipment locally. It would be interesting to conduct visual/face
processing studies through the web and include eye tracking data with this
library. Granted the accuracy is low compared to specialized equipment and
predictions are highly variable, other contributors could enhance its
accuracy. I suspect that the variability in shape, size, and structure of
human faces contributed greatly to the high variance/low resolution
predictions issue. Maybe categorizing the user's face shape a priori would
enhance predictions. That is, the library would first estimate the "type of
face" and use those data to inform the eye tracking. This might not make sense
--just a thought.

------
stephen
Just curious, but does anyone who pays attention to this space/technology know
if eye tracking (with commodity-ish hardware) has gotten to the point of
"replacing the mouse"?

E.g. at the OS level, the pointer basically follows your gaze.

~~~
trishume
Eyes don't point a things we look at as accurate as a mouse does (they only
aim within the fovea, about 1 degree angular, corresponding to about 1cm on
the screen) and eye trackers are often far worse.

This summer I built a system that uses slight head movements for refinement,
allowing hands free mousing as fast as a trackpad. However, eye trackers good
enough for the system to be pleasant cost $10000+. I have ideas that would
allow it to work with a $100 eye tracker, but even then it has the downside
that the computer vision for your mouse will peg one core.

~~~
yetanotherjosh
As someone who hates mice and hopes one day to have this sort of tech, I'm
trying to understand what you mean by "Eyes don't point at things we look at
as accurate as a mouse does" \- how is it that I can choose to "look at" an
individual pixel on my screen, or an individual serif on a letter, and then
choose another one far less than 1cm away, and my eye focus shifts to it? That
shift is not detectable by a camera?

(Edit update: by "detectable" I mean theoretically detectable by a really good
camera and software. In other words, it seems you are arguing the tech is
impossible even theoretically due to some aspect of biology. Am I following
you correctly there? Thanks)

~~~
jacobevelyn
I'm also not an expert, but I suspect a confounding problem is the fact that
they eyes never stop moving, even when it feels to us like we're looking at a
single point:
[https://en.wikipedia.org/wiki/Eye_movement#Saccades](https://en.wikipedia.org/wiki/Eye_movement#Saccades)

------
CommanderData
I think this could be useful with Digital Signage. My client uses HTML to
display content and is run in browser which makes it easy to design and deploy
content. He says one of the challenges is getting seamless user interactivity
with his browser based digital signage.

This coupled with web camera recognizing hand gestures e.g.
[http://hadi.io/gest.js/](http://hadi.io/gest.js/) could help grabbing user
interaction inside his apps without too much hassle but right now its a janky
solution compared to a real native kinect setup (I don't think its even
possible to effectively connect to kinect in browser).

~~~
j_s
Someday soon USB devices will be accessible, thanks to the Chromebook. Not
sure if that will help accessing the Kinect.

[https://wicg.github.io/webusb/](https://wicg.github.io/webusb/)

------
Vespasian
Well after I figured out that I could use clicks to calibrate the tracker it
was somewhat accurate (ca. 100-200 px), however hte variance was quite large.

I think it's a really nice idea, however not yet precise enough to work for
such tasks as user studies

~~~
falcolas
100px gives you around 70 zones in your average desktop window. More, when you
consider shared hits between zones. That's enough to do basic heat maps of a
web site (such as for ad impression tracking), coarse navigation, or
integration into things like editors/IDEs.

~~~
grenoire
I was thinking about ad impressions as well; but God forbid websites are ever
able to use my camera for eye tracking by bypassing the webcam permissions (or
through the removal of in the future? I expect anything when it comes to
content monetisation).

~~~
falcolas
Or simply refusing access to the content without access to the camera
(thinking of Forbe's ill-conceived content blocking).

~~~
artursapek
I will laugh my ass off if it ever comes to this. Hopefully I never have to
come back to this comment and weep.

------
kevinsimper
It sadly did not work for me either, even though I tried "calibrating" it with
moving my mouse around the screen, but when i moved the mouse out of the
window it was just as bad as before.

~~~
danneu
You need to click.

Open the game and follow the mouse with your eyes while you click each
8-directional edge of the game's viewport and then click the center.

Now you can let go of the mouse (don't need to move it out of the game) and
the game should follow your gaze.

This demo is impressive. It's accurate enough for me to isolate the orange
ball from the cluster.

On a website, the calibration could be done in a modal overlay. "Look at this
dot while clicking it wherever it appears". After then Nth click, the modal
goes away and the user lands on the website ready to gaze.

------
jhanna
Nice work, well done.. one of the best one's we've seen !!!

I am one of the founders at xLabs and we too have been working on self
calibrating real time interactive webcam eye tracking for a number of and have
a number of demos and commercial products.

We'd love your feedback.. check out
[http://xlabsgaze.com/](http://xlabsgaze.com/) (core tech) and our first
commercial product [https://eyesdecide.com](https://eyesdecide.com) (Design
effectiveness and usabilty testing SaaS).

Our just check out one of our video here
[https://www.youtube.com/watch?v=kSnAfJWAhtE](https://www.youtube.com/watch?v=kSnAfJWAhtE)

Keep up the good work.

------
amelius
This could be useful to predict where the user will click next, so the website
can preload data, with the result that user waiting time is reduced.

~~~
dimal
I assume it's meant for in-house user testing, not on the live web.

------
dyarosla
100px accuracy. I thought eye tracking could do better than that in general?
At 100px accuracy, WebGazer is more useful to advertisers than it is to
consumers, as I can't see any consumer-friendly applications that would be
useful with that precision. If that's the case, adoption won't grow. Are there
other comparable eye tracking libraries out now?

~~~
falcolas
> as I can't see any consumer-friendly applications that would be useful with
> that precision

How about:

\- switching focus between windows

\- application selection a'la mission-control

\- coarse navigation through a document via a sidebar

\- real-time inventory management in a HTML5 game

\- Selective zoom of embedded images

Even more fun: Atom is run inside a web browser. Imagine what you could do
with eye tracking within an IDE.

~~~
Morgawr
>Imagine what you could do with eye tracking within an IDE.

You provided really good cases for the rest, but for this I can't quite
picture a need (other than maybe scrolling, which is arguably faster using vim
movements already). Do you have any ideas?

~~~
danneu
Maybe it could be used for modal keybindings if your gaze changes modes e.g.
focus between the panes of an IDE. hjkl moves the cursor when looking at the
code, but it traverses the file hierarchy when looking at the sidebar, and
enumerates tabs when looking at widgets.

Kind of like how keys are reused in Vim yet have similar meanings across each
mode. You don't have to remember so many unique keybindings.

Some editors let you switch between panes with keybindings, but once you have
more than a few or some in weird shapes, you end up needing tmux's `C-b q`
pane selector. Seems like gaze could replace most of that.

------
nikki-9696
As a user, I find this creepy as f __k. I would love to know how to disable
this sort of "feature" on my phone.

~~~
huskyr
If you have an Apple device you don't need to disable it: webcam video streams
(aka getUserMedia()) isn't supported on Safari. On other browsers you always
need to give permission before the camera is switched on.

~~~
nikki-9696
Oh, great to know, thanks!

------
rawnlq
It didn't work for me until I did the following:

\- Zoom in so that my gaze is actually looking at distinct parts of the
physical screen

\- Look at the cursor while _clicking_ a few times to train it

\- Take off my glasses

\- Move way closer to the camera and hold relatively still

It was a lot of hassle to get it to work but after it did work it was a lot of
fun!

~~~
metasean
> Take off my glasses

I was wondering if that was the problem for me, but I'm so near-sighted that
if I were close enough to actually see the cursor, there's no way the camera
could see my eyes to track them. :-(

------
fredleblanc
In the demo, I had to click on the screen a couple of times to get the dot to
start moving around. Then, it seemed to want to follow my mouse quite a bit as
I moved and clicked.

I was able to greatly move the red dot when I say closer to the webcam, really
opened my eyes, and moved left to right, but the action was flipped for me —
looking left moved the dot right, and vice versa. (I'm using an Apple Cinema
Display… I've had issues before where it automatically flips the screen around
in some cases.)

Although not very accurate, it's certainly an interesting experiment.

~~~
WalterSear
Did you calibrate it at all?

~~~
kevinsimper
How do you calibrate it?

~~~
WalterSear
The instructions were on the page. Get it looking at your face properly, and
then look directly at the mouse cursor and click it, on spots around the page.
Make sure you don't move your head, just your eyes.

~~~
uptown
Maybe the site could detect if people have read the instructions.

------
kecks
It didn't work for me at all; the red dot never appeared on my screen, though
the face-outline in the preview image did fit my face quite well (when it
wasn't detecting my chin as my mouth).

~~~
lazyjeff
Hi, one of the authors here. It was probably not very clear on our part, but
it's meant to train the eye tracking model as naturally you use the website.
So it takes advantage of your interactions over time, like if you're using
Gmail.

So if you're doing the demo on the blank page, click in a couple of places
around the screen (while looking there as you would normally), and you should
start seeing the prediction.

~~~
kecks
Oh, thanks, it worked now!

------
polartx
I'm on a thunderbolt display (with integrated camera) sitting back about 3
feet. After 5 or so clicks against the corner regions, tracking became
surprisingly accurate. I'm impressed!

------
benlower
Very interesting project. Was excited to try this but the quality of the
tracking is not there yet. The face detection was off most of the time unless
I positioned my camera _just_ right. Then the eye tracking was no good. But
they are only using a webcam and doing this right (see
[http://www.tobii.com/xperience/](http://www.tobii.com/xperience/)) requires
multiple IR emitters and an IR camera.

------
alistproducer2
It didn't work that well with the camera on my HP 8470p but great idea and I'm
sure it took a lot of time and effort. I appreciate the effort.

------
octopitus
Interesting. Just curious, but does this library scalable for a facial
recognition system? Eg: Authentication system.

------
33a
Cool idea and opens up a lot of interesting applications. Unfortunately, the
demos don't work reliably (on my machine the tracking was completely off the
wall).

------
swang
cool trick but we all know who will end up using this the most:

advertisers. another way for them to "gauge" their reach and brand appeal. or
something.

literally i feel like every new thing that comes to the browser, a marketer at
a company thinks, 'how can we exploit this to put more junk into the face of
other people?'

i am not against advertising, i just think most of them have no limits in
terms of what they'll exploit at the expense of their own users.

~~~
untog
Well, advertisers couldn't use this without asking permission for a user's
webcam, which would be a non-starter. I choose the less cynical path and say
this could be tremendously useful for remote user testing.

~~~
arsenerei
Remote usability testing would be fantastic. This could be useful for local
testing as well. Having a library or tool that could generically sit in front
of a site to show where users look when taking action X sounds beneficial to
me.

I can also imagine on a lighter note this being used for games where you look
to where you want to move a character (maybe blink twice for confirmation).

------
samfisher83
How many people are going to be hesitant to allow people to use their webcams?

I can't get it to work, but even if I could I would be very hesitant to use
it.

------
reubensutton
I wonder if anyone has experimented with using something like this to make an
incredibly inexpensive assistive technology interface

------
transpy
Could this be useful to monitor user sentiment and correlate it with further
data from applications/media?

------
Javadavinci
This is interesting. I don't see many practical implications though.

------
goatic
pretty cool!

