

Show HN: My weekend project - Chessboard picture recognition - matthiasb
http://codebazaar.blogspot.com/2011/08/chess-board-recognition-project-part-1.html

======
gfodor
It seems like you can avoid having to recognize the pieces altogether if you
are taking photos of each move during a match, since they have a known
starting position.

Edit: what I mean is that in this case the problem becomes answering "does
this place on the board have a piece on it" not "which piece is on this place
on the board." Once you know which places on the board have any pieces
whatsoever you can simply infer between moves which pieces moved where.

------
patio11
You might look into the academic literature for OCR, which this will reduce
to. (Most AI problems are "How can I apply human intelligence to cleverly
reduce this problem from something which appears to require intelligence to
something which actually only requires well-understood math?")

I think the punchline is likely to be that you end up taking dot products of
feature vectors generated by library code and writing fairly little chess-
related code. Sorry to burst your bubble if you thought this was going to be
very sexy. On the plus side, it will work, and if you're as good at
programming as you appear to be you're almost certain to succeed on this.

P.S. I hope you like taking photos because odds are you're going to get the
opportunity to take _lots_.

~~~
matthiasb
Thanks patio13! OCR and AI actually seem very sexy to me:D If I make a mobile
app out of this, I will get my users to take pictures for me! hehe!

~~~
patio13
Hey I'm patio13 and I have nothing to do with it :-)

~~~
matthiasb
lol. sorry but you get the credit for patio11:D hehe

------
swanson
I don't know anything about "chess theory" but are there some squares that
only certain pieces can get to? I don't think it's possible for a Knight to
ever be on some squares, so maybe you could use logic like that to narrow down
the piece detection.

It may be easier to just track the game state from the start instead of trying
to identify the position of the board at a random snapshot in time.

Cool project though, seems like a good intro to image processing.

~~~
matthiasb
Thanks for leaving a comment.

"It may be easier to just track the game state from the start instead of
trying to identify the position of the board at a random snapshot in time."

This is certainly true! But the original idea was to take a picture (probably
from a mobile phone) and continue the game online. Taking a picture after each
most is a lot of work :D

~~~
danohuiginn
attach a camera (or phone) to the clock. It takes a picture whenever you hit
the clock (i.e. after every move), and records the move history. You can stop
scribbling it on those little pads.

------
meatsock
"For example, the program is able to determine whether each square is white or
black by calculating its average color and comparing it to the black and white
colors"

I bet you could get away with determining the color of a corner square and
assuming the rest.

~~~
matthiasb
Thanks for the feedback. I did not think about this.

Some of the square edges actually belongs to the surroundings squares because
the cut is not 100% accurate and the chess piece are not always position at
the center of the square but I could get around that.

The way this algorithm works is simple. I compare the RGB values for each
pixel of a square to the black color and white color. Black is Red=0, Green=0,
Blue=0 and white is Red=255, Green=255 and Blue=255. I add up the 3 RGB values
for a pixel, if it is closer to 0 and 255*3, I consider the pixel is mostly
black. I keep a counter of black pixels and another for white pixels.

Currently, the algo is just comparing the two counters. If I found most of the
pixel were black, I consider it is a black square.

However, I could improve it as following: \- > 90% of black pixels: empty
black square or black square with black piece \- > 90% of white pixels: empty
white square or white square with white piece \- other white: square with a
piece of the opposite color.

For the 3rd case, I might need to implement your idea.

~~~
max99x
Most pieces are positioned in the middle of a square. A simple way to
recognize the color of the piece and its square is to do a flood fill with a
bit of tolerance (say 32/256) from the center pixel. The area covered by the
flood fill will only contain the pixels making up the piece, while the area
outside the flood fill will have pixels the color of the square. Reflections
might cause a bit of noise, but a slight gaussian blur should be enough to get
over that.

------
Game_Ender
I think your best bet is to feed the the segmented chess board squares into a
machine learning algorithm. You would have one which classifies squares as
empty or not, and another which recognizes chess pieces. The difficult part is
building up all the examples to do the training, picking the right algorithms
to use (SVM, logistic regression, etc), and then transforming the raw data
into the proper form for the algorithm.

I did find a very recent paper on a chess playing robot which has some good
starting points:
[http://www.cs.washington.edu/ai/Mobile_Robotics/postscripts/...](http://www.cs.washington.edu/ai/Mobile_Robotics/postscripts/chess-
icra-11.pdf)

~~~
matthiasb
Thanks for the link. It will help!

------
nostromo
Hmm, interesting project.

If you're ok with helping the computer out a bit, you could put a very small
matrix bar code (think QR Code) on top of each piece. Since there are 6 chess
pieces and 2 colors, you could have a code for each piece in just a 2x2 matrix
(4 dots / bits).

Otherwise, it seems like it may be very difficult, as I'm not even sure of
what some of the black on black pieces are myself (bottom right).

~~~
shib71
If you can get games in machine readable format, it may be possible to develop
a ML approach that figures out the most likely identifications based on
relative positions and as much information as image processing provides.

------
awaz
I have worked in machine vision projects in past. It's not going to be easy to
recognize the pieces as is; I suggest that you use 2D pieces for chess. Even
better, use some unique design like this one:
[http://www.chess.com/forum/view/general/experimental-
chess-s...](http://www.chess.com/forum/view/general/experimental-chess-set-
design)

~~~
shahan
Agree that it will be tricky to recognize the pieces, but what's the point in
doing it with 2D pieces? The app would be (slightly) more exciting if it could
help you record the state of any game anywhere in less than 5 seconds -- and
allow you to continue playing online.

~~~
awaz
I suggested 2D if he wanted to build a system for himself. May be multi-angle
snap-shots or a video capture could be processed for capturing conventional
chess boards. But even that would not work for all chess-boards. I have come
across some pieces that are different to distinguish.

------
binarymax
I may have a good solution:

Take _two_ photos. One at the starting position and one at the current
position you'd like to save. You don't need to take them in order (take the
current position, reset the board, and take the other photo). This way you
know what the pieces are from the starting position, then use an algo to match
based on that.

Also some good hints for most standard sets while checking your current
position...both kings are always on the board and they are usually the
tallest. there is never more than one queen per side, and its usually a close
height to the king, but has a circular crown. Rooks are circular on top,
bishops come to a point, if there are more than two pawns per side that gives
them away. Knights could be singled out by eliminating the rest.

Good luck!

------
codeswimmer
Very nicely done! Really appreciate the links to your reference material in
the blog, saves me (and other noobs) from having to dig around and discover it
ourselves. I'll definitely be keeping track of your progress. Good luck, dude!

~~~
matthiasb
Thank you :D

------
shahan
Don't see the use of this as an app, but fuck the use. If the goal is fun and
learning, I think it's worth going all in and do object recognition using,
say, SIFT vectors (see Wikipedia). Just to see if it works.

It's easy to find SIFT implementations, but you'll have to do some work to
construct a training set of images for each type of chess piece. An elegant
solution will probably detect the type of piece independent of its color, or
the color of the square.

This might be ridiculously easy, or not work at all.

------
vynch
yes..like the other hackers are saying....you have to deal with this problem
like a poker player tries to guess his opponents cards....so the first step is
to create the piece expectation function...which will basically calculate the
probability that a piece is (say knight) at square x,y given the probabilty
expectation functions of all the other squares..one way to create
probablilities for all squares would be to use a combination of dynamic
programming and neural networks...you will have to create a dynamic
programming algorithm that sort of trains itself(by storing data rather than
changing its code!!) do better and better as you tell it how right or wrong it
was....so the topics you are looking to cover in the next exercise (assuming
tht u have already done some reading on image processing) are basic
probability theory,dynamic programming and neural nets...those are my two
cents on the problem.

------
matthiasb
If one day, my program is able to recognize the pieces on this chessboard... I
will happy:D
[http://2.bp.blogspot.com/-TgamS2dnAls/TjvpV3YuA0I/AAAAAAAAAI...](http://2.bp.blogspot.com/-TgamS2dnAls/TjvpV3YuA0I/AAAAAAAAAI4/y6XESlITAN8/s1600/287621_2266996910302_1109935532_32714307_1964814_o%25281%2529.jpg)

~~~
sebinatx
Haha! I recognize this board :) Good luck, my friend!

------
imurray
Here is a related project on turning an image of a Go board into a computer
readable file understood by most GUIs for playing Go:
<http://www.inference.phy.cam.ac.uk/cjb/image2sgf.html>

~~~
matthiasb
Thanks. Go seems to be easier. I was also thinking to fallback to domino if I
can completed the project for a chessboard.

------
ebzlo
For figuring out chess pieces, maybe determining the light source and looking
for light/dark patterns in certain areas? The problem is, from top view, even
a human would have a difficult time figuring out if a piece was a bishop or
pawn.

~~~
matthiasb
This is true. I may need to work on a collection of pictures or even a video.

~~~
danohuiginn
Get hold of one of those automated chess boards that move pieces around by
magnets under the board. Set it to replay old games, and film or photograph
it. That should give you unlimited photographs of known positions.

~~~
matthiasb
I did not know about these boards. Good to know. Thanks for the tip.

------
mitko
similar successful project of a robot playing chess naturally:

<http://www.youtube.com/watch?v=OZ0Hx04KFCg>

~~~
matthiasb
I just found out about this. Another YC user left a comment with a link to a
paper about Gambit the robot:
[http://www.cs.washington.edu/ai/Mobile_Robotics/postscripts/...](http://www.cs.washington.edu/ai/Mobile_Robotics/postscripts/chess-
icra-11.pdf)

