

Show HN: Flutter - Control Spotify or iTunes Using Gestures thru Webcam - mehuln
http://flutter.io

======
tsunamifury
Amazing, works very well.

I assume it is looking for 5 fingers. I noticed if i folded my thumb in, it
did not recognize the gesture so maybe an edge case where people missing
fingers can't use this.

Also when I scratched my face briefly it did recognize it... funny.

I would strongly recommend trying to build a web demo somewhere down the line.
Even if it requires flash to get the webcam working, I'm sure your conversions
would go through the roof if you could show how well it worked instantly on
your landing page.

Small note -- the blurring of the image is distracting rather than focusing.
Down the line maybe think about a better landing page image.

~~~
shearn89
Just tried this: 4 fingers with my thumb tucked across my palm worked fine for
me.

------
mehuln
We hated using the keyboard and mouse for controlling music in the background
or watching movies on our computers from a few feet away. So we decided to
solve this problem by using hand gestures to control music and videos via the
built-in webcam in your computer.

Please download and try it. Do let us know what you think in your comments
below or by emailing us at feedback [at] flutter.io.

Currently, it works for Mac OSX 10.6 and up.

~~~
wmw
is the download broken?

~~~
mehuln
hi wmw, it should work as long as you have Mac. If you are on other OS then it
will ask you to leave your email address for future invite.

Are you on Mac? Can you tell us what happened?

~~~
wmw
after typing in my email and name i got this: The link that you have given is
not valid. Please verify that you have copied the link completely. This
incident will be reported.

------
joshmlewis
I hardly ever get excited after downloading something, maybe a, wow that's
cool, but when I installed this (which was very easy) and it just worked, I
actually clapped and said 'this is fucking awesome.'

~~~
ajju
I had the exact same feeling. You see these gesture controls in movies and
demos and they seem great, but it's _truly_ exciting when you get to use it
yourself.

Love it!

~~~
navneetdalal
Ajju thanks for the encouragement. We are glad you liked it. If you have more
usage ideas or feature requests, do let us know.

~~~
pokoleo
Thought that this was great - how about a next/previous track gesture, and
being able to change the frequency that it checks for the hand? (it looks like
it's time based)

------
apu
I believe one of the co-founders is Navneet Dalal, who, along with Bill
Triggs, developed the "Histograms of Oriented Gradients" (HOG) features that
are widely used across all of computer vision†. Thus, I'm guessing that
there's some serious vision in use for this app, which is pretty awesome!

(My apologies to the other co-founders, who are likely just as talented!)

† <http://en.wikipedia.org/wiki/Histogram_of_oriented_gradients>

~~~
mehuln
Yes, he is the brains behind this!

------
lukev
This is awesome.

However, music control isn't a pain point for me. What I _would_ pay for is a
start/pause gesture for Netflix/Hulu/Amazon Video. Worst thing about TV on a
computer.

~~~
mehuln
Luke - that's a great point. We're working on it and it will be in the next
version!

~~~
toomuchtodo
Once you guys have that down, take my money and I'll bolt an iMac to my wall.

------
twakefield
I just ask my very non-technical girlfriend to come see this new app I
downloaded. I showed her flutter.io working in action. She shrieked and said,
"that's awesome, how do I get it?". That's a very good sign for you
guys...nice job.

~~~
mehuln
Haha - that's great. Thanks!!

------
ihodes
Though I know there's a chance you'd not be willing to share the information,
I'd love to learn a little bit about how you built this product.

It appears as though you're using ML to recognize the gestures—anything else
you could share would be fascinating. I'll be sending in a job application, as
well!

~~~
mehuln
You're right. We use image processing, computer vision & machine learning to
detect Flutters. Our entire tech stack is custom written as we are solving
some quite challenging & difficult AI problems.

------
paul9290
Interesting/cool project, though I do wonder how you plan on keeping it simple
when you expand and add additional actions?

The UX of controlling media using the Xbox Kinect leaves a lot to be desired -
it's cumbersome.

I've had my mac mini connected to my tv for awhile and am very interested in
the Internet TV space in re: what will become the standard way we control such
technology? Will it be thru gestures, airplay on iphone/ipad type technology,
voice driven or a mix of all?

For many years I've been using a wireless gyration mouse/keyboard (demo of
Internet TV setup <http://www.youtube.com/watch?v=l-IryK4wsUM>) and recently I
started to use Chrome's voice search to quickly bring things up especially if
my wireless keyboard isn't reachable.

After seeing this and my years of enjoying the Internet on my TV I'm thinking
a few short gestures like this app uses and mix of voice driven actions might
provide the best UX.

Based on the views and comments of video I link to above people want the
Internet on their TV and all it provides. We just haven't figured out the best
UX for it.

~~~
mehuln
Paul - that is a great observation and feedback. We believe that future will
be about us using touch, voice, and gesture all three depending on context and
circumstances. We do plan to keep it simple, and we have some great ideas for
it.

------
Roritharr
This is amazingly useful... just not on the pc.

Implement it for Android and you can charge me 30€ so i can easily control the
mp3 player in my car.

~~~
joshmlewis
I think you'd have to take a slightly different approach for use in the
car..unless you had a mount for your phone on the windshield.

------
bdunbar
"The music swirled and dived for a moment. Another voice broke in, presumably
Halfrunt. He said "Vell, Zapbod's just zis guy, you know?" but got no further
because an electric pencil flew across the cabin and through the radio's
on/off-sensitive airspace. Zaphod turned and glared at Trillian -- she had
thrown the pencil. "

HHGTTG - Douglas Adams

------
plasma
Flutter in action:
[http://www.youtube.com/watch?v=IxsGgW6sQHI&feature=youtu...](http://www.youtube.com/watch?v=IxsGgW6sQHI&feature=youtu.be&a)

~~~
cellularmitosis
Expected reaction, having just a blurred-out image on your landing page:
"Sure, whatever, it probably doesn't work that well...".

Expected reaction, having a video demonstration of the product on your landing
page: "THIS IS FRICKIN' AWESOME!".

Please, please, but a video on your landing page. You will thank yourself
after checking your conversion rate.

~~~
mehuln
Hi, thanks so much for letting us know. We will put the video. The one on
YouTube is old, but we will do it very soon. thanks!!

------
nileshk
This is very cool. It worked perfectly even in the dimly lit room I am sitting
in.

What I've been wanting and have thought about developing myself is a universal
play/pause control that pauses whatever you happen to be listening to at the
moment, whether it be a desktop client like iTunes, Spotify, Rdio, etc, or if
it is something running in a web browser like Pandora, Soundcloud, Youtube,
etc.

This app takes a step towards this in the way it handles iTunes and Spotify at
the same time, by controlling whichever app had focus last.

I imagine for web browser control a browser extension would necessary. And
ideally a plugin architecture would exist to allow 3rd parties to add support
for additional apps.

~~~
goronbjorn
A different but functionally similar idea would be an automatic pausing of all
other services when you start playing one (i.e. when I start playing a Youtube
video, automatically pause iTunes).

~~~
Zirro
There needs to be a way to override those behaviours though, as there are
situations where the audio of a YouTube video (for example) isn't very
interesting, and you'd rather listen to music. Lack of background-playing in
the iOS YouTube-app is still an annoyance to me.

------
devin
Really cool. I was just playing with it and my girlfriend walked over to ask
me a question. I held up my hand to my computer as if to say "Hold on,
computer."

It was a bit surreal. Very cool. Can't wait for more gestures or a generalized
way to program my own gestures. I would love to be able to gesture "Switch to
iTerm.app" by doing an OK sign or something. Sign language letters maybe?

Anyway, thanks. Really enjoy it already, can't wait for more. Also, I would
pay money for this. If it had all of the play/pause/forward/back functionality
I think I'd give you 4.99 if it was accurate. Just sayin'.

~~~
mark_integerdsv
My thoughts almost exactly on the programmable front... like Better Touch Tool
but rather No touch Tool, although the latter name does retain the locker-room
hilarity of the former.

Perhaps a gesture to bring up the app switching ribbon and a gesture to
reference the item (numbered from left to right maybe.)

I'd also like to have the option to calibrate it, my eyes are painfully
photosensitive and I tend to work in very dimly-lit conditions.

------
siavosh
What's nice about this is you don't need to switch screens to pause the music
if you find yourself writing/coding and need some quiet when you're in the
zone. Granted you can do this with the mac play/pause physical buttons but
that would require your eyes to look down and pause your thought process.

This happened to me just now after I installed Flutter, and it seemed to me
that physical gestures didn't interrupt my higher brain functions. Of course
when I noticed this it made me interrupt my work and come to HN to leave this
comment. The irony.

~~~
mehuln
Glad you noticed it. We want you to never think about controlling your music
at all. This is one of the things I love too... we listen to music in a
background, so controlling it should be in the background too.

------
andrewpi
Flutter seems to keep the camera on 100% of the time which prevents apps like
Skype from using the camera for video chats. I'm not sure if there is a good
workaround for that?

~~~
mehuln
Andrew - great observation. On top of your menubar, there's Flutter icon.
Click on it, and it will give you simple flutter on and flutter off
capability.

~~~
toomuchtodo
Feature Request: Designate a gesture to turn Flutter off.

~~~
mehuln
Great idea. Although, you probably will have to turn it on manually.

------
cmelbye
This is fantastic! I can't wait until it's extended with more gestures for
next track, etc, but this initial version is so awesome! Nice work.

------
basil
There is another, more fully-featured app called FluidTunes which I used quite
a while ago. Not sure if it still works on the latest version of iTunes:
<http://majicjungle.com/fluidtunes.html>

------
mdanger
This is awesome, but one question/suggestion: when a user is going through the
"How to Use Flutter" tutorial, do the gestures work then, or only once the
tutorial is closed? I kept trying to do it, but there wasn't a response until
I closed the tutorial.

~~~
mehuln
Only when tutorial is done and you're using iTunes or Spotify. First time you
bring up either of the apps, it will notify you through top menu bar that
Flutter is controlling it now.

Once that happens, you can control in the background or even if it is
minimized.

------
intended
One of the few occasions that I wish I had a mac.

Few questions: Aren't you entering a field which has patents covering gesture
controls on media exactly like this?

A comparative advantage of mice and keyboards is that a minor amount of
energy, (but large amounts of co-ordination) are leveraged into larger
changes. Wont this limit the number of the gesture controls you can
incorporate?

(OTOH - if you had enough fidelity, you would just need a dummy keyboard. You
would just have to map finger movements with key presses. you could just
create a webcam of your own as well.)

------
hedgehog
Very cool although with CPU usage at about 30% on my Air I won't be able to
use it much. If you can figure out a way to drop average CPU usage down to
reduce heat & power usage that would be a big deal.

~~~
mehuln
Hi Hedgehog - thats a great point. We thought of it, and when you're not doing
a gesture, CPU usage should drop to around 2-3%. Let us know if it doesn't. In
our test, average CPU usage was around 5%.

~~~
hedgehog
Interesting. It does seem to drop to about 5% if I'm not in front using the
computer and not in front of it but otherwise floats between about 7% and 20%.
This is on a MacBookAir4,2, based on a few minutes of testing it seems like
it's about 65% additional power consumption when the machine is otherwise
mostly idle. I'm actually impressed though that you got power consumption as
low as it is. Photo Booth consumes about 6.5% so I'd guess that's the lower
limit for Flutter, maybe default to disabling Flutter while the computer is on
battery?

------
doubleconfess
Soooooo. Doesn't this open up a whole new realm of privacy concerns? Who cares
if Path has my contacts list, I'm more concerned that you guys are watching me
surf for porn!

~~~
mehuln
Legit concern. This is why we've built an app w/o a server component, so
everything happens on your machine. We do not capture anything from your
webcam. It all happens on your machine. On top of it, there's a turn off
capability which will allow you to turn it off for time being.

~~~
savrajsingh
I presume the connection opened to AWS on boot is just for tracking purposes,
yes? :)

~~~
mehuln
Yes, that is correct.

------
shriphani
Hi guys. Since there seems to be an ML component, I was wondering how this was
built. Do you start with a basic hypothesis and see if it holds for some
training set and then refine it or right off the bat go with a generic
classifier like SVM (with some features in mind) or something like that and
then tweak it? Also, how did you choose the dataset for this ?

This project is very cool. Congratulations and good luck!

------
sikhnerd
Would be nice if we could get this to work with other media players by
emulating the media keys (really want to use this with vlc on my macbook)

~~~
mehuln
We would include VLC very shortly. On the list.

------
ttruong
I was blown away when I saw a demo of Flutter a couple of months ago. Their
Spotify and iTunes integration only scratches the surface of all the gestures
that they supported in that demo. I can't wait for them to add some additional
gestures, such as skipping to the next track, and PC support so that I can get
rid of my keyboard with the dedicated music player controls.

------
jigneshrmehta
This must be amazing app to download as it is clear from the feedback
received. Please don't forget that we are a large enough community who uses
windows. Especially in India. We are already excited even before you have our
version. the kind of capabilities it gives, would be amazing to download.
Congrats in advance to the team.

------
jrheard
Fantastic app, does a fantastic job of recognizing my palm - unfortunately, it
seems to also really like my face. After the tenth or so time that it had
paused my music without me raising a hand, I realized that every time I turn
my right cheek to the camera, Flutter pauses my music. Which is a little
troublesome :)

~~~
mehuln
This is a great feedback. If you don't mind, take a picture and bluetooth and
send it to us at Feedback [at] flutter.io

------
notpmocd
This is fantastic - absolutely the most satisfying method of interaction with
an app that I've experienced in a long time. Expand the gesture set to include
skip forward/back and it's a winning consumer app. Personally, I'd love (and
would pay for) a library allowing me to add gesture control to my own apps.

------
sarbogast
Now that's what I call a very awesome MVP!

------
_ankit_
Amazing, the single gesture is _very_ well implemented.

Can't wait to swish my hands around to skip songs! :) And once you have that,
you should build a futuristic minority report kind of a demo video ;)

Also, if you're looking to hire, you should definitely put a "Team" page up on
your site.

------
harigov
I did a similar hand gesture recognition software to control image viewer as
part of my computer vision course project. It was really fun to do. I believe,
presentations and lectures are the places where this kind of technology really
makes a lot of difference. Good luck with your startup!

------
mark_integerdsv
Blown away.

How far are you guys from full Minority Report-ization of my Mac?

I mentioned in another post that I tend to work in very dimly lit spaces due
to painfully photosensitive eyes. A non-mandatory calibration dialogue would
probably be a handy thing for folks like me.

------
Galactica
Neat and simple, but as an end user I would use this a lot more on my TV than
on my MAC/PC. Samsung built in cam below.

[http://www.youtube.com/watch?v=Szr5hGU3EK0&feature=relat...](http://www.youtube.com/watch?v=Szr5hGU3EK0&feature=related)

------
alexobenauer
Wow - I originally didn't download because I've used something similar before
that worked only kind of well.

This works fantastically and is usable enough for daily use as a simple way to
control iTunes or Spotify. Well done!

I'll look forward to more gestures.

------
PaulAnunda
I saw Samsung demoing some similar technology during SXSW (using a kinect that
was hooked up to a mac mini and being displayed on one of their LED tv's), and
it didn't work anywhere near as well. Congrats.

------
tar
Please note that this seems to be available for Mac users only right now.

------
raccoonone
Looks awesome, can't wait for a Linux version, so that I can try it out!

------
guynamedloren
Works great once I realized that I couldn't be so close to the camera, but
does anybody have any tips on disabling the green iSight LED on my macbook pro
(short of covering it)?

~~~
mehuln
Would you want that as a feature? Keep camera on but turn off the light?
Hmm...

~~~
wesleyzhao
Sounds like being able to do that could be a huge security/privacy concern...

------
mehuln
A user just created a video for us - Thank You So Much!! :)
<http://www.youtube.com/watch?v=lhnlUMHUlNc>

------
instakill
Quick note. Your site assumes I am using Chrome - FF doesn't save downloads in
the bottom left hand corner. Easy for me to figure out, impossible for someone
like my mom.

------
vipinsahu
Brilliant work! truly awesome . Shall i expect more in future for WEB like
this ? <http://depthjs.media.mit.edu/>

------
iz_mani
Hey guys this is some really cool stuff. If you can integrate this with
browser gestures to scroll, refresh, and move back and forward on pages. This
would be awesome.

------
huskyr
I _so_ hope this is an art project and in a few days we'll have a website of
people making funny gestures to the camera to pause and play Justin Bieber
songs ;)

------
jmilinovich
In case anyone was wondering, I can confirm "mirror stops" work: try facing
your laptop towards a mirror and doing your gestures from behind the computer-
wow!

------
kranner
On Lion it seems to register itself automatically in startup items. There is
no Preferences or Settings window to change this either.

Very awesome overall.

------
patrickod
This works incredibly well. I wasn't sure how accurate this could be without
the kinect-style depth sensor but it's surprisingly accurate. Great work guys!

------
lzy
So this is Mac only?

~~~
mehuln
Yes, at the moment. We will add support for other OS very soon.

------
crikli
I've been sitting here for like five minutes just starting and stopping. About
every third time I laugh and say (out loud) "that's so awesome."

------
fosk
It would be interesting to know what piece of technology, or framework,
they're using for recognizing gestures using the webcam.

~~~
savrajsingh
I'm not sure, but this looks like a good starting point?
<http://www.movesinstitute.org/~kolsch/HandVu/HandVu.html>

------
shravan
It seems like the Minority Report future of gesture based control is not that
far off. Great app!

------
sainib
You have built something people will want to talk about but why limit to
spotify or iTunes :)

~~~
mehuln
Hi, we are not limiting it to that. We want to add more to the list. It just
takes time... we're still a small 3 people team. Know any great
engineers/designers who can help us execute faster? :)

------
conroy
Please allow me to control a presentation in Keynote. I've wanted this feature
for so long.

------
TorbjornLunde
Looks very interesting.

The download link seems be down though (internal server error).

~~~
mehuln
Hi. Can you please try again? It just worked for me? If you still get an error
then please send email to Feedback [at] flutter.io an we will send you diff
link. Thanks so much. M

~~~
TorbjornLunde
Seems to be an encoding error of sorts. I have a norwegian character in my
name (Ø), i used switched it for O and the download worked fine.

The app works great, really cool!

------
ashishtoshniwal
This is so freakishly awesome !!!

------
instakill
This is so good that I'm putting VLC away to use iTunes to listen to my music
library.

------
payalg
issue is that even when i am not controlling the playlist and randomly moving
my hand it recognizes it as gesture... seems like lot needs to be done to
differentiate between noise and actual gesture. anyways nice try.

------
savrajsingh
Great app. Looking forward to support for more gestures!

------
younata
That's amazing.

I... wow.

Take my money.

------
kitarpyor
Wonderful job guys!!! very very impressive

------
berberous
Desperately want a skip gesture.

~~~
mehuln
We will try to get in out ASAP!

------
avner
i imagine you guys are already working on the skip/back functions. Awesome
initiative!

~~~
mehuln
Yes, idea is to allow you to listen to music or watch movies without touching
keyboard or mouse.

------
peterdelahunty
Makes me feel like a jedi !!!

------
Technomento
This is amazing!!

------
dmils4
Great job guys!

------
playhard
Brilliant work!

------
jzhou
this is sick, like @zigfu

------
zackattack
It would be cool if this also let me clap as a gesture. And it would be cool
if you sold an arduino kit that let me hook up my light switches.

------
shingen
Congrats guys, this is very cool.

------
sgtsugarfingers
Surveillance in the name of convenience has gone too far. I can see a future
where companies offer services as wrappers around visually analyzing people's
behavior in their own homes. They will sell that data just as they do now.
That's not a future I encourage.

~~~
mehuln
That's a great point. This is why we've built an app w/o a server component,
so everything happens on your machine.

