

Hacking on Side Projects: The Pool Ball Tracker - tomblomfield
http://blog.gocardless.com/post/34568593614/hacking-on-side-projects-the-pool-ball-tracker

======
weaksauce
One approach that worked well for us when we tried to do this for an automated
labyrinth project was to only operate on one color channel and template match
an ideal ball while updating the template based on lighting conditions. Our
project was slower moving though so the ball didn't deform quite as much. Our
webcam was terrible so there was probably more visual distortion than I seem
to recall.

Edit: also in our case we used a region of interest that moved with the ball
position to cut down on the computation.

~~~
hcm
Interesting! Constraining tracking to regions of interest is definitely on our
list of things to try.

~~~
weaksauce
You might have good luck with the background removal techniques because your
background is stationary. In our case the camera was stationary but the board
constantly moved so that tactic was not very useful in our case.

Edit: also something to consider is using a kalman filter to help predict and
smooth out noisy predictions. We did that and it helped considerably.

------
arctangent
Neat project.

More fun pool table shenanigans:
<http://www.obscuradigital.com/products/detail/cuelight/>

IIRC Obscura were selling that for $100k+. If the current business model
doesn't pan out then I guess GoCardless could always pivot into that market
instead :-)

~~~
nathan_f77
Oh wow, that would be an amazing addition to the GoCardless project. I look
forward to playing their open source code, but first I need a pool table :)

------
kyberias
This is the kind of stuff I want to see in Hacker News!

------
abeh
Huge pool fan here: Put this in a mobile device that I can mount from any
angle and record all my games anywhere - I would pay a lot more than $0.99 for
it!

------
alexhawdon
Very similar to my undergraduate research project based on tracking bubbles
moving through the air. I also started with the Hough transform but found that
things broke down quickly when the targets were moving at speed, then switched
to segementation and morphological operations which worked really nicely when
set up correctly.

Great application - looking forward to the future posts :)

------
pk2200
For version 2, please get this running in Google Glass and give me realtime
aiming assistance. :)

~~~
gknoy
They could project a light on the table based on expected ball movement, to
show bounces off of the bumpers or other balls. That would be tremendously
cool as a learning aid.

------
georgespencer
Very cool. Is there a reason why you aren't tracking the black in the video?

~~~
hcm
The black proved slightly harder to track than the yellow and red balls, as it
didn't have a particularly distinctive hue. The first step towards tracking
the black was filtering out everything except the green surface, which is
demonstrated in the video. Unfortunately we ran out of time before getting the
black tracked.

~~~
w0utert
I'm wondering whether you could apply an edge-detect filter to your input
frames and combine it with the hue-based detection. I'd assume that
irrespective of the ball color, an edge detect + posterize would give you a
video frame with clear spots where the balls are, which should be relatively
easy to find. I remember from the image analysis courses I took in college
that spot detection is a very common operation in automatic analysis of
medical images, and hence extensively researched. After you know where each
ball is you could convert the frame to YUV and use the UV (chroma) channels to
detect the ball colors.

~~~
weaksauce
Edge detection algorithms are fairly slow and noisy from what I found. To do
this in realtime you have to have fast algorithms or cut down on the
information to process over.

------
mathewsanders
This is cool! I'm sure that their is a huge business opportunity here too (I
don't watch sports but sure that their must be entire channels dedicated to
snooker/pool etc) you could make games a lot more exciting to watch with VR
replays of interesting shots etc (check similar companies for virtual
sportscasting for America's cup and off-road racing :)

~~~
robmil
It's already been done I'm afraid — certainly on snooker coverage in the UK,
they frequently do VR demonstrations of possible alternative shots, VR views
of different angles on the table, etc.

------
stcredzero
How about an app that tracks the ball positions and gives advice on the
"leave?" Knowing how to optimize for your own fallibility and minimize
positional advantages for your opponent if you miss and maximize them for when
your aim is good is actually the most important part of the game.

~~~
alexhawdon
F'ckin love this idea!

So it learns your strengths/weaknesses then when you're playing uses this
information to evaluate each possible shot based on the predicted final
position of the ball and how useful this would be to the opponent and your
chances of success...

One possible difficulty would be the learning phase - it would have to be able
to know your intentions when you take a shot, which would be difficult to
guess automatically. I believe it might be billiards where players have to say
which ball they're going for - you might have to build that in...

~~~
stcredzero
Just picking from the 3 easiest shots would take care of most of it.

------
Itaxpica
This is very, VERY cool... but I can't help but think that an Arduino and some
color sensors might be easier. Or some small RFID stickers and basic readers
as long as the stickers didn't mess with the weight/spin of the balls, since
color sensors might have problems with striped balls depending on spin.

Actually, figuring out how to do this without a webcam sounds like a pretty
snazzy hurricane project.

------
j3kuntz
For a really fast consumer webcam, look at the PS3 Eyetoy
(<http://www.amazon.com/PlayStation-Eye-3/dp/B000VTQ3LU>).

There are very stable linux drivers out there for it and you can get up to
120fps. For a past project, my team was able to get up past 60FPS using this
camera.

------
jrockway
I've been meaning to do this for chess, which seems like it would be much
easier than pool. The checkerboard pattern of the board is rather simple to
distinguish, and you don't need to visually distinguish individual pieces,
just what squares the pieces are on.

~~~
aqme28
Don't you need to distinguish individual pieces? It's probably hard to discern
chess shapes from above (unless they're specially designed for this).

You could "cheat" and track the game's progress, but that wouldn't work for
pawn promotions.

~~~
jrockway
I don't mind manually entering the chosen promotion. (I've seen someone
promote to a knight once, but only because he was being a jackass. Promoting
to Queen would have been checkmate.)

------
grecy
I wonder how much more effort it would be to predict where a ball is going to
end up once it's moving....

For some reason I'm thinking of those guys who did this with a roulette
wheel.. but I'm not sure how that will be handy for pool.

~~~
georgespencer
It wouldn't be handy but it'd be fucking cool. If you know the velocity and
trajectory of the ball then on an unobstructed shot you could instantly
display the updated position of the ball after it has been struck. The
difficulty would be gauging things like back spin which aren't (I'm guessing)
observable by the camera, but which have a drastic impact on where the balls
end up.

~~~
gknoy
Great point! I wonder if using predictions of ball movement might make it
easier to recognize them, as well.

~~~
apu
Yes, predictions can greatly improve recognition performance. Not only can you
focus the bulk of your computations on the expected region of interest, but
you can also potentially reason about appearance (e.g., if you know the frame-
rate/integration time of the camera and expected speed of the ball, you could
get estimates about how much blur to expect). You could also better handle
difficult cases, such as when balls strike each other (e.g., simple
conservation of momentum estimates should greatly constraint possible
hypotheses for ball positions).

------
thornofmight
This is awesome!

------
anandagarwaal
nice.

