
Turning a MacBook into a Touchscreen with $1 of Hardware (2018) - soegaard
https://www.anishathalye.com/2018/04/03/macbook-touchscreen/
======
duxup
That is amazing.

I really wouldn't have expected this to work but once you read about it.... of
course it would.

This is one of those things that while there maybe wasn't a 'problem' shows
how knowing a few things critical, you can really solve problems / come up
with solutions that nobody would have ever thought of (well I wouldn't
have...).

~~~
macspoofing
>I really wouldn't have expected this to work but once you read about it....
of course it would.

It's a fun DIY project and you can get it _almost_ working. I suspect edge
cases (stemming from different lighting conditions and imperfect finger
detection) would render this method unworkable in general.

~~~
kreetx
I wonder if this has ever been tentatively tested by the laptop makers,
perhaps even by Apple. Surely thought about -- but tested?

~~~
dharma1
There are plenty of Windows laptops with a touchscreen. In Apple's case MacOS
isn't really designed for touch

~~~
kreetx
No-no - I mean using the webcam and a mirror. I.e there would be a little
mirror that would fold out of the screen.

~~~
that_jojo
I think his point was that it would be a bit silly to put this kind of
solution into a product -- which is, in a word: jank -- when capacitive
touchscreens are already a tried and true solution in production devices.

~~~
tomp
I often think “hand-pointing” interfaces have unexplored / unrealised
potential... like, who would want a touch TV screen?! Sure, you could have the
touch UX for the TV on your phone... but even better would be if you would be
able to just point to the TV and _MAGIC_. The biggest problem I see is that
humans aren’t that good at pointing - we can only “aim” with one eye, keeping
both eyes open messes it up.

~~~
Groxx
I've often wondered what having one of these on a laptop, tracking space above
the keyboard / in front of the screen, would be like:
[https://www.leapmotion.com/](https://www.leapmotion.com/)

Gestural stuff is nice when it's transparent and guess-able. Sadly not often
the case, but when it is it's pretty magical.

~~~
mncharity
> on a laptop, tracking space above the keyboard

I tried doing this with the internal webcam, using a clip-on fisheye lens, and
mirrors. And eventually punted, using the bare webcam only for head tracking,
and adding usb cameras on sticks perched on the laptop screen. With more
sticks to protect the usb sockets from the cables. And lots of gaff tape.

> leapmotion

Leap Motion has finally been acquired, so the future of the product is
unclear. And it's Windows-only (the older and even cruftier version supporting
linux, doesn't do background rejection, and so can't be used pointing down at
a keyboard). But it has apis, so you can export the data. My fuzzy impression
is it's not quite good enough for surface touch events, but it's ok-ish for
pose and gestures. When the poses don't have a lot of occlusion. And the
device is perched on a stick in front of your face.

> Gestural stuff is nice when it's transparent and guess-able. Sadly not often
> the case

I fuzzily recall some old system (Lisp Machine?) as having a status bar with a
little picture of a mouse, and telling what its buttons would do if pressed.
And a key impact of VR/AR is having more and cheaper UI real estate to work
with. So always showing what gestures are currently available, and what they
do, should become feasible.

Even on a generic laptop screen, DIYed for 3D, it seems you might put such
secondary information semitransparently above the screen plane. And sort of
focus through it to work. Making the overlay merely annoying, rather than
intolerable.

But when it all works, yeah, magical. Briefly magical. The future is already
here... it just has a painfully low duty cycle. And ghastly overhead.

------
cletus
So I was expecting one of those sensors that can turn a TV into a touchscreen
with like an IR sensor or something so I checked this wondering how they were
so cheap.

I was delighted to see this is so much better. Figuring out touch events from
the reflection is really cool. Well done.

~~~
Aperocky
What they did not factor in was the cost of the software.

You can make junkyard bound stuff do amazing things when you program them
right, but you pay for that programming or time spent.

~~~
nullandvoid
The software is a "one time cost" in quotes due to maintenance etc but talking
in broad strokes which wouldn't impact the claim of $1 touch

Or to put it another way you wouldnt invent a new way to desalinate seawater
and then say the cost is $10/litre instead of 2 cent because of r&d on the
device you made..

~~~
musingsole
> invent a new way to desalinate seawater and then say the cost is $10/litre

Depends on the marketing strategy, most likely. But I like GP's point about
undervaluing programmer time. It is non-free and everyone down to the
programmers themselves tend to grossly underestimate its cost.

Humans are kind of expensive to keep alive and happy enough to program
willingly

/That last statement had more caveats than I wanted...

~~~
tripzilch
It really depends on their intention, whether they consider that part of the
work DIY or not.

If I paint my wall, I would generally say it costs the price of the paint, not
whatever I'd have to pay to have someone do it for me.

Or what about the people who tinker with their own smart home solutions with
arduinos and raspberries. If you factor in the price of work/research/coding,
it would be very hard to justify not buying an off-the-shelf solution instead.

~~~
musingsole
> If I paint my wall, I would generally say it costs the price of the paint

That's the mentality I'm challenging here. Mentally you don't account for the
cost of your time, but from an outside perspective, you're spending it all the
same.

That you have to factor in the price of labor with DIY electronics is the main
reason you don't see more of those easy electronics projects available as some
off-the-shelf contained product (where's my IoT API-accessible thermometer for
$10!!!???).

------
shadofx
Next step is to gauge the amount of blood underneath the fingernail to tell
the amount of pressure being applied.

~~~
macca321
I had an idea years ago using a webcam and the 'blood under fingernail' to
have a trackpadless-trackpad (i.e. same as this project but on your table).

Maybe some time-rich person will fork it and do so.

------
_hardwaregeek
This is exactly what a hackathon project should be. Delightful, innovative and
scoped small enough that they produce results without too much bullshit. Not
the usual "world changing" hacks that end up abandoned the moment the event
concludes. If only events encouraged more hacks like these.

~~~
sli
The last hackathon I attended, the winners just built something from an
Instructable that existed well before the hackathon.

My friends and I built a time machine radio (i.e. you tune it to a decade,
with a knob and everything, and radio news from the decade was included if we
could find it) and won best hardware hack, so it wasn't a total loss. Had a
blast either way.

~~~
ergothus
> The last hackathon I attended, the winners....

(Not a comment on you, but you poked the button)

While my hackathon experience is limited, I think any attempt to crown a
"winner" reduces the experience. Of course, I am too sensitive because
everyone can just enjoy it...but as soon as there's a winner there starts to
be rules that people try to game, the are added restrictions and limits that
reduce what is being created, etc. And creating is the point.

~~~
setr
However, you do want to show off the more interesting/higher quality projects,
to encourage further work, and get the whole thing jiving (people _enjoy_
competition; it oils the machines)

The problem I think is that judges themselves are incompetent/careless — when
you promote a “bad” winner, you just discourage everyone from trying, because
they realize there is no value to this kind of promotion; its bullshit.

I had a similar experience in my university — some kind of business project
idea competition; we submitted my lab’s current project, and there were other
interesting projects involved who I would have been happy to lose to.

Instead I lost to the girl with the contentless idea for edible spoons for
developing countries... an idea I’d seen repeatedly in recent news cycles
because some NGO in India was doing it for the last three years.

Five minutes of googling would have discovered this, but instead we simply
ignored the competition since.

Attending itself was valuable, along with the half-attempt at winning — we
clarified the project and ideas, and had an excuse to clean things up — so it
would have been a good exercise to do yearly (and probably for the other
projects involved), and the competition was healthy.

But it just takes an incompetent judge to spoil the whole thing.

There’s nothing wrong with people trying to game the competition, its a
natural function of competing. But the judges are meant to operate as experts,
and an expert should have (some) ability to discern bullshit from something
real.

~~~
_hardwaregeek
Agreed. I wish hackathon judges actually understood how to question and
analyze projects critically. I attended probably one of the most prestigious
and well regarded college hackathons, where the grand prize was won by a
project that did speech to text, then text to sign language. What nobody
pointed out or acknowledged, was that __deaf people can read __. But since the
project was done on a HoloLens the judges got swept away by the fancy AR and
awarded them the prize.

I've become extremely skeptical of the hackathon model in general. It doesn't
really produce good projects or products, it encourages an unsustainable style
of working and it's a terrible place to recruit good developers. It can
encourage people to build stuff, but only monkey patched, bursting with
buzzwords, overly flashy projects that are thrown away at a moment's notice.

~~~
setr
>What nobody pointed out or acknowledged, was that deaf people can read

I actually did something similarly stupid in college; for an embedded
development class, my team needed a final project. I proposed a hat for the
blind, with distance sensors and buzzers on the inside to inform distance from
the walls and such. We also used black conductive thread on a black hat, so it
was impossible to work with except for the girl who threaded it in the first
place.

Easily the professor's favorite project, and I guess he still uses it as an
example for future projects (and at some point he had some team extending it
into some kind backpack?)

Later a friend working at a retail store selling products for the disabled
contacted me about the project; apparently his boss was interested.

What no one acknowledged is that blind people are not walking into walls...
the cane is far more practical, efficient and effective tool than this thing
could ever be.

I had meant it to be a joke.

~~~
svachalek
Rebrand this as a device for people to not waste precious time looking where
they're walking, and it's a gold mine.

~~~
squiggleblaz
"Ever worried about those people who are reading their phone while walking
around? Now you can randomly put this hat on their head."

The best thing about it is, the people who need and want the device aren't the
ones walking around reading their phone, it's the ones who can't help but mind
someone else's business - and this merely feeds their need rather than
satisfies it. So there will be an never-ending market of unsatisfied
busybodies.

------
innerop
[https://www.playosmo.com/en/](https://www.playosmo.com/en/) <\-- this also
uses a mirror to refocus the built-in camera but on another surface, not the
screen.

------
bitL
Hmm, given camera is on top, computing homography and rectifying the
perspective image would give fairly good precision around the camera, falling
off quickly with distance, likely not being very useful at the bottom of the
screen. Also, it's likely sensitive to both ambient light and the dominant
light on the screen, in dark rooms with dark themes it might be pretty off, as
most computer vision algorithms need constant tuning for changing conditions.

~~~
mbreese
Maybe you could add an IR LED that could be observed by the webcam, but not in
the visible spectrum. I’m not sure if the MacBook webcam includes an IR filter
or not. It wouldn't help the precision, but might make the camera less
susceptible to changes in ambient light.

~~~
tigershark
Even 5$ webcam include a IR filter, the image quality without is pretty awful,
so I’m pretty sure that apple webcams have it.

~~~
kevinsundar
I recently shined a remote control into my late 2013 macbook pro's camera and
I totally saw the ir light. Not saying that you're wrong, I don't know if
there are filters but only for certain specific wavelengths or something like
that.

~~~
mkl
Remote controls are pretty bright IR, so it could be just that it only blocks
up to a certain brightness.

~~~
kevinsundar
Makes sense. Direct IR light is probably harder to block.

------
pcr910303
Wow, this is a great idea! Looking at the video, looks like the latency isn't
great; what kind of problems should be solved for this to have a ~100ms
latency?

Rewriting it in C/C++/Rust instead of Python would be one... Any ideas?

~~~
sdegutis
Slightly off topic, but C/C++/Rust are always the go-to languages when talking
about high performance. People love the speed of C but it's so bare-bones that
they move to C++, or it's too dangerous so they move to Rust. But those are
much too complicated, so people came up with Zig and Nim and D instead, but
those are very _far_ from C, even though they're not _as far_ as C++ and Rust.
So I'm wondering, are there any languages _almost as close to C_ but
_slightly_ safer and _slightly_ less bare-bones? Because I like C a lot, I
think it's a great language with an excellent simplicity, except for some
slight rough edges.

~~~
eclat
Crystal? Hasn't reached 1.0 though.

~~~
snazz
I think Crystal is garbage-collected, right?

~~~
rafaelvasco
Yeah, the whole language is designed around it. Impossible to remove without
rewriting it from scratch: [https://github.com/crystal-
lang/crystal/wiki/FAQ#language-x-...](https://github.com/crystal-
lang/crystal/wiki/FAQ#language-x-has-feature-y-why-dont-you-have-such-feature)

------
moondev
Wow! This is really creative. Gets my gears turning on all the other
applications for using opencv as an interface!

------
aidos
I love this. When I was young I saw a hack for an Amstrad which added light
pen support. If I recall it effectively synced with the scans on the crt which
tells you your coordinates. I wanted to build it so much as a kid, but never
could. It’s the sort of thing that really fired my imagination and helped me
fall in love with computers.

------
cs02rm0
I wonder what the field of view is like. The demos only show it being used
towards the middle of the lower half of the screen - can the camera really see
the top left or right corners?

~~~
dfee
From the conclusion:

> With some simple modifications such as a higher resolution webcam (ours was
> 480p) and a curved mirror that allows the webcam to capture the entire
> screen, Sistine could become a practical low-cost touchscreen system.

------
lifeformed
The hovering aspect of this makes it something quite special, a feature normal
touchscreens don't have. I imagine css ::hover elements would feel nice with
this.

------
melling
Maybe next someone will build a “camera keyboard“ for my iPad. Turn any
surface into a keyboard with a camera and machine vision.

~~~
smeyer
Do you mean similar to a projection keyboard[0] but without actually
projecting a keyboard onto the surface and using the integrated camera?

[0]
[https://en.wikipedia.org/wiki/Projection_keyboard](https://en.wikipedia.org/wiki/Projection_keyboard)

~~~
johnday
Ooh, how about this?

1\. Point a camera at a surface (be it a piece of paper or whatever).

2\. Write/print a word on that paper and use CV/OCR to parse it.

3\. Detect when a person has pressed on that word and exec the command
specified by the text.

Instant macro pad?

~~~
cgriswald
4\. Scatter bunches of 'rm -rf /' pieces of paper at your local wifi hotspot.

~~~
martin_a
I see, you thought the same as I did. :-D

------
eric_khun
Is anyone found a stylus that works with the macbook's trackpad? The usecase
is to be able to draw some diagrams during video calls/screencasst.

Trying to avoid the wacom/tablet path to keep it lightweight :)

~~~
elSidCampeador
I bought a case in India for my Kindle Paperwhite which came with a stylus
[1]. That stylus works, and works quite well with my 2015 MBP.

[1] - [http://www.proelite.co.in/Amazon-Kindle/All-Amazon-
Kindle-10...](http://www.proelite.co.in/Amazon-Kindle/All-Amazon-Kindle-10th-
Generation-2019-Slim-Smart-Flip-case-Cover--Deep-Blue--with-Stylus-Pen--Not-
Fit-Paperwhite--id-3984513.html)

------
bomanbot
Very neat! I am wondering whether it could also recognize multiple fingers and
thus sort of emulate multitouch gestures, although my guess is latency is not
great with one finger already, so this might not work well.

Als interested to see how it works with different lighting and how dirty the
screen can get before recognition fails because the reflection is too
„muddled“ ;)

------
atoav
One issue I imagine is coverage. How do you get decent horizontal resolution
on top and on bottom of the screen?

Either you have a mirror that converts your webcam into ultra wide angle so
you cover the top edge of the screen (which you pay for by poor resolution on
the bottom) or you decide not to cover the whole top area (which is where most
window control elements are).

The video action all happens on the lower half of the screen for a reason.

A solution would be some sort of orthographic lense, that basically splits the
sensor into multiple distributed lenses to avoid the FOV problems. This
however would be not trivial to get right without serious tools.

~~~
regularfry
Use a pinball, not a flat mirror, and map back to something like a flat plane
before processing. The registration is a bit harder, but not by an intractable
amount.

------
jankotek
Long time ago there was an ad PC vs Mac. PC guy had usb camera tapped on top
of his head, claiming how PC are entering new modern age. Mac had camera build
in. This kind of feels similar.

------
zoom6628
Pure brilliance. While many commenters seem to focus on computer applications
this could be very viable solution for industrial environments for touch
controls.

------
guelo
That is great!

On the topic of touchscreen Macbooks, I've been using my Mac a lot less since
I've spent a lot of time on a touchscreen Windows laptop. It is super handy! A
lot of times when using the Mac I'll absentmindedly drag something on the
screen only to be disappointed. Apple's resistance to the idea reminds me of
their stubbornness on the single button mouse. Admitting you're wrong is not a
weakness.

~~~
grecy
What hardware do you have? All the touchscreen windows laptops I've tried have
the responsiveness and accuracy of a touchscreen from 1990.

~~~
shawnz
Resistive touchscreens which were dominant in 1990 typically have higher
accuracy than the capacitive touchscreens of today. But they have terrible
usability because you have to apply pressure to the screen to use them.

------
redthrowaway
And they called it Sistine, which flabbergastingly shows that coming up with a
good name for something is indeed possible.

------
TheRealPomax
This probably needs [2018] in the title?

------
ph33t
Stunning out of the box thinking. Fantastic idea and implementation. Color me
impressed.

------
chickenchaser
This is super interesting, I'm about to undergo a small DIY smart mirror
project and wanted touchscreen capabilities but all the hardware solutions
seemed impractical and expensive. This might be a super easy way to make that
work.

------
Aperocky
It's a good idea and great implementation.

However I do strongly feel that touchscreen is the opposite to productivity,
maybe usability to the end users, but definitely not what developers
need/want.

~~~
s_y_n_t_a_x
It helps older users who have trouble with the mouse/touchpad.

~~~
sbradford26
A touch screen is definitely a more approachable method to interact with
computers for people who are not as tech literate. My grandma had a computer
for years and only kind of used it. Then she got an ipad from a relative and
she uses everyday, from casting netflix to her chromecast to reading ebooks
she rented from the library.

I may not prefer touch screens but they can be a complete game changer for
some users.

~~~
donarb
There is nothing wrong with touch screens when they are horizontal like a
tablet. Vertical touch screens like on laptops or desktops are not ergonomic,
and especially so for the elderly.

~~~
WorldMaker
Entirely horizontal touch screens have the same problems as
trackpads/keyboard/mice (or piano playing) that they can have in terms of
wrist strain/RSIs (including carpal tunnel), and sometimes elbow movement
strain/RSIs (including things like "tennis elbow").

Vertical touch screens trade the more traditional wrist/elbow strain/RSIs for
other arm/elbow/shoulder strain/RSIs (including things like "gorilla arm").

Given the nature of repetitive stress, the growing consensus seems to be __not
__that one orientation is better than the other, but that a _range_ of
orientations, and a mixture of movement types / input methods is better. For
instance, move your arm some to touch a vertical screen, to mix things up from
mouse movements and keyboard movements.

------
newaccoutnas
This is great. I notice it's Python 2, could this be ported to Python 3 easily
or some inherent issue in upstream deps?

~~~
jpeanuts
While I understand the desire for a unified Python ecosystem as much as anyone
- this kind of pettifogging is not going to convince anyone to switch to Py3.

Consider if the authors of the article had used (for example) Ruby - you
probably wouldn't ask why they didn't do it in Python3, so why do you do it if
they write it in Python2?

~~~
codetrotter
I think they asked in a respectful manner, so I don’t see anything wrong with
their comment really.

Python 2 will be EOLed soon.

[https://pythonclock.org/](https://pythonclock.org/)

4 months 25 days 5 hours left until Python 2 is retired. After that point it
will no longer be maintained by the Python project.

Of course Linux distros and companies will keep Python 2 running for years to
come because of all of the tools that are Python 2 only.

New projects ought to be written in Python 3. However, in this case my guess
would be that OP probably used Python 2.7 because macOS ships with Python 2.7
preinstalled.

So if we could, Apple are the ones that we ought to speak to. Convince Apple
to ship macOS with Python 3 pre-installed.

------
stackzero
Incredible. And here I thought a good assortment of lollies at the local
corner store was the best $1 could do

------
thomasfl
I hope this hack get’s so popular that the folks at Apple finally makes a
MacBook with a proper touch screen.

------
innocentoldguy
I normally hate touchscreen computers, but this project and its implementation
are really cool.

------
tiku
Could we just add some shiny window tint foil to normal laptop screens to
achieve this effect?

~~~
mkl
This is a normal laptop screen. What do you mean?

------
188201
This project makes laptop vendors selling touchscreen laptop with premium
price cry.

------
ethanpil
Curious what kind of CPU usage the OpenCV code requires on the backend...

~~~
mighty_bander
There is some well-informed speculation about latency up-thread, but I've
tinkered with OpenCV from time to time. Specifically, I used ROS and OpenCV to
do face detection on a netbook in about 2012. That worked out pretty well - it
would track multiple faces in realtime. Edge detection is a different
algorithm, but isn't super likely to be the origin of the latency issues
they're having. It's just a threshold check over a window of a few (e.g. 3 or
5) pixels, multiplied by the dimensions of the image.

------
broken_symlink
What sorts of improvements could you do if you used a stylus instead?

------
stcredzero
_The basic principle behind Sistine is simple. Surfaces viewed from an angle
tend to look shiny, and you can tell if a finger is touching the surface by
checking if it’s touching its own reflection._

This is especially true of MacBook screens!

------
daveheq
Yay now their software can see everything on your screen.

~~~
notimetorelax
As opposed to a driver that may have greater permissions?

------
nsxwolf
This could be extended for multitouch too, yeah?

~~~
robotresearcher
Yes, but it would have a possibly annoying limitation of occlusion making the
multi-touch fail at some positions. Fingers very close to the camera would
appear huge and occlude everything else.

------
chrisweekly
This is really f'ing cool. Nice work!!

------
quickthrower2
Is this something he should have patented?

~~~
mkl
The main idea was probably patented decades ago. There have been optical touch
screens around for ages (that use multiple cameras for good multitouch, and
edge IR illumination to make detection more reliable and precise).

------
kizer
That... is a _hack_! Awesome :D

------
shireboy
Now, that is just smart.

------
rammy1234
whoa! amazing. out of box thinking

------
AzzieElbab
Brilliant

------
cvaidya1986
Well done!

------
ideasRgood
Prototypes are cool, but I think this is dead.

Anything that requires tons of visual calibration is serious work. Both for
the programmer and the computer.

It worked for a planned demonstration, but I imagine this would be impossible
for Mass production. And I wonder if the users found it useful due to latency
and accuracy problems.

~~~
Liquix
The authors didn't publish thinking "this is going to rock the world, it's
ready for mass production"... More along the lines "hey fellow hackers, we
found a way to emulate a touch screen at 1% of the price".

~~~
ideasRgood
But if the latency and accuracy isn't great, did they solve anything?

~~~
empath75
They weren't trying to solve anything. They wanted to see if they could make
it work, and they did.

------
punnerud
DUPE
[https://news.ycombinator.com/item?id=16745998](https://news.ycombinator.com/item?id=16745998)

~~~
kusha
>Are reposts ok?

>If a story has had significant attention in the last year or so, we kill
reposts as duplicates. If not, a small number of reposts is ok.

That post is from over a year ago, and didn't get much traction.

~~~
minimaxir
Although it was indeed posted over a year ago, 239 points is certainly
traction.

~~~
soegaard
Missed the old post. Thought reposts were rejected automatically (like on
Reddit).

~~~
minimaxir
FWIW it does, although it's fussy and only works on posts in the last few
days.

tbh Reddit's repost detector causes more problems than it solves (it still
triggers if the original post was removed).

