First you have the quality of optics. Most computers have very small cameras that are low resolution and prone to noise in situations without ideal lighting.
That makes eyes hard to capture as a whole.
Then you need to figure out eye direction. Eyes flit around a lot (saccade) but you could perhaps smooth it out. But pupils are hard to see anyway through glasses. You better hope people wear large glasses with skinny frames and don’t suffer from very poor eyesight or astigmatism, both which lead to high refractions.
There are actually good products for this like tobi (sp?) etc where you can wear prescription lenses and have IR tracking for your eyes.
but even the , even if you get over the technical issues there’s the UX issue. How do you account for something getting your users attention without changing the input focus there? Let’s say they’re listening to music and a track changes, showing a notification.
And even if you figure all of that out, there’s the privacy angle. People don’t like being monitored constantly.
I've been trying to flit my eyes around the screen to get a sense for how fast you'd need to track. Definitely seems pretty fast! At least sample 30Hz I'd bet. I could see how doing all the image processing that quick might be tricky w/out custom hardware (even w/out the glasses problem you mentioned).
For the UX, I was imagining you have to press a button to instruct the computer that "Hey I'd like for you to move my cursor via eye-tracking". That way the cursor only moves when you want it to (same as today w/ a mouse) and isn't constantly moving around when you look around. Press down to have it move cursor to eye-tracked position and stop when you release.
Could possibly decompose the space bar to have that space for that button. Like have 3 mouse buttons where the right side of space bar is: (a) Track my eye movement while I press down and stop when I release, (b) left-click, (c) right click. Then you don't have to leave home row on your keyboard.
Or add one of those IBM Thinkpad mouse knubs somewhere on a keyboard and use those as mouse buttons instead of a mouse itself.
So here’s the other rub with using the keyboard to enter the command. Most people , even experienced touch typists, will flit their eyes towards the thing they’re trying to interact with.
Therein lies a big part of the problem with eye based interaction. Our brains move our eyes for a lot of different tasks, saccade to get a constant read of your scene (eyes have very poor resolving power so need to move a lot), they also signify what you’re thinking (there’s a lot of studies in neurolonguistics about eye direction signalling how you’re thinking, but at a base level, you tend to look up or away when you’re pondering).
Anyway not to say it can’t be done. But it’s a fascinating domain at the cross section of UX, technology and neural science.
For what it’s worth, there are VR headsets with dedicated eye trackers built in (PSVR2, certain Vive Pro models, Varjo etc..) and there have been cameras from Canon (in the film days even!) that used eye tracking for autofocus targets.
It’ll be interesting to see how things shape up. Meta have their big keynote on Tuesday where the Quest Pro / Cambria is meant to have eye tracking sensors.
Maybe you could place 3-4 30-60FPS cheapish CMOS cams around the edge of the screen and stagger their frame captures? You'd get different angles (better for detecting eye vectors) and increase the sampling rate.
(Can't reply to your other comment for some reason)
Why is the IR part of the spectrum better for the cameras? Is it because if I take an image of my eye and look at the IR part of the spectrum is it just easier to see parts of my eye that determine where it's looking?
First you have the quality of optics. Most computers have very small cameras that are low resolution and prone to noise in situations without ideal lighting.
That makes eyes hard to capture as a whole.
Then you need to figure out eye direction. Eyes flit around a lot (saccade) but you could perhaps smooth it out. But pupils are hard to see anyway through glasses. You better hope people wear large glasses with skinny frames and don’t suffer from very poor eyesight or astigmatism, both which lead to high refractions.
There are actually good products for this like tobi (sp?) etc where you can wear prescription lenses and have IR tracking for your eyes.
but even the , even if you get over the technical issues there’s the UX issue. How do you account for something getting your users attention without changing the input focus there? Let’s say they’re listening to music and a track changes, showing a notification.
And even if you figure all of that out, there’s the privacy angle. People don’t like being monitored constantly.