
Show HN: Scannable Chess Scoresheets - indy7500
https://www.reinechess.com/
======
parimarjan
This is definitely a real problem for some people - a chess organizer friend
of mine recently spent 4-5 hours entering games from a scoresheet into the
computer every night. This solution is pretty nice - but it is possible to
imagine that such a solution might work directly on horribly written
scoresheets, using handwriting recognition + the kind of constraints mentioned
by indy7500 - that would be pretty amazing from both a technical, and
usefulness, standpoint.

~~~
indy7500
(I am OP) I thought about doing it that way but found that handwriting
recognition was pretty weak as of now and wanted to try something solid. The
reason is, usually, handwriting recognition would take into account the
following characters (it doesn't recognize individual characters at a time, it
looks at a database of handwritten whole words). That would be the ideal if
done well, of course.

~~~
doodpants
I have no experience with handwriting recognition, but intuitively it seems
like the facts that:

a) the grammar for chess move notation is fairly limited; and

b) each move must be valid relative to the current game state

can be used as contextual clues to resolve ambiguous characters. Of course,
this means you can't use an off-the-shelf handwriting recognition package that
recognizes whole words like you describe, but requires a custom per-character
recognizer. (But then, isn't that what you're doing already?)

~~~
indy7500
Now that I think about it, you're right. I don't know what the accuracy would
be like, but it's certainly possible and worth a try.

------
bambax
> _No recording errors: if it 's "Bg4" and you write "Bg5", we can't help :(_

You can help by checking the legality of the current move (which you seem to
already do) and also the rest of the game?

~~~
indy7500
The way we check legality and adjust is with the character recognition. We use
the second highest and third highest confidence characters. So, if you write a
5 and the model recognizes it as a 5 and then a 6 and 7, but really it's a 4,
we can't help. This is because we'd add moves exponentially every time we add
that many combinations

~~~
theli0nheart
I don't get it. If Bg5 is an illegal move then the recognition is obviously
wrong and you can prune it (I would think?).

Who am I to talk though, I'm not the one programming this thing. That's just
what my gut tells me. :)

~~~
indy7500
Don't say that, you're definitely more experienced than me :) I'll try to
clarify, but I could be wrong. This is from the chess.com article on it here:
[https://www.chess.com/blog/ReineChess/scannable-
scoresheets-...](https://www.chess.com/blog/ReineChess/scannable-scoresheets-
free)

The model actually gives us a confidence for each possibility. It might say,
N-90%, K-10%, B/R/Q-0% for one of the boxes. Here, we look at how confident
our character recognizer is on each character. If the PGN is invalid, then we
know some character was recognized incorrectly. We look at the low-confidence
characters and change them to the next highest confidence prediction, checking
to see which combination of changes delivers a valid PGN. For example,
consider the string of moves 1. d4 d2. Black can't play d2 on move 1! But we
look at the next most likely predictions, perhaps d8 for white and d5 for
black. These are the combinations:

d4 d2

d4 d5

d8 d2

d8 d5

Only the second one is valid, so we choose it and continue ahead.

If you literally wrote "d4 d2," then it's a lot less likely that the correct
digit instead of 2 will be in the top 3 predictions.

~~~
theli0nheart
This all makes really good sense. For now I think this is a workable strategy,
but I bet you can do better (take that as a challenge ;) ).

Anyways, this is really cool, and smart idea. I'll be printing these out and
bringing them to my next tourney. Excited to see how it goes! How do I send
you direct feedback?

~~~
indy7500
We currently don't store scoresheets when uploaded to our website. Would you
be able to send the pictures to us at info@reinechess.com? You could add
comments about how it went (or not, since we can just run it and see for
ourselves). If you think that might be a hassle I could quickly add a feedback
form on the website. Also, I'd encourage that you try it once before the
tourney with the restrictions—we haven't tested on very varied handwriting.
Oh, and thanks so much for offering feedback!

P.S. Did you have something in mind for improving the postprocessing / trying
out different iterations? Or was it just a challenge :D

~~~
fr0sty
Is there a reason you don't store the images of submitted games? Acquiring
real-world training data to test changes with would be very useful. Perhaps
make it an opt-in feature people can select when uploading a game so people
are "offering" you this information to you rather than you just "taking" it.

One way to choose between two legal candidate moves would be to look forward
for the next move by the candidate pieces and see which ones make sense given.
If you are trying to choose between e6 and e5, for instance, look forward and
see what the next e pawn move is. Plenty of other ways to go about this, of
course.

~~~
indy7500
No, not a clear reason other than wanted to get the product out earlier. We'll
work on an opt-in, that does sound the best.

Wouldn't other pieces affect this though? For example, what if the move was e6
and the next e-pawn move was e7. What if a piece was already there, Ne7
(wrongly recognized)? I thought about slowly running through it like this, but
it felt like I kept running into problems which are obvious for us to
recognize but I wasn't sure how to implement it computationally

------
gk1
This is neat. For anyone wondering why not just record the moves on your
phone: Digital devices are not allowed at over-the-board tournament games.

~~~
gmiller123456
As others have pointed out, there are digital scoresheets. But further,
digital devices are not banned at OTB tournaments, most players will have a
cell phone, and might even text or take calls. It's only in high stakes
tournaments that you'll see cell phones banned, but even then it's only for
the players. It's still perfectly legal, and extremely common for the game to
be video recorded.

~~~
radiator
I am not aware of what you mean by "high stakes tournaments". All official
tournaments, even when only amateurs play, actually ban cell phones since
about 15 years ago.

[https://www.fide.com/FIDE/handbook/Anti%20Cheating%20Guideli...](https://www.fide.com/FIDE/handbook/Anti%20Cheating%20Guidelines.pdf)

11.3.b. During play, a player is forbidden to have a mobile phone and/or other
device capable of processing or transmitting chess analysis in the playing
venue. If it is evident that a player brought such a device into the playing
venue, he shall lose the game. The opponent shall win. The rules of a
competition may specify a different, less severe, penalty.

~~~
munk-a
That rule only seems to apply to devices the player controls themselves, so if
the tournament had a recording device or if both players jointly sponsored a
recording device it doesn't appear it would trigger that rule.

I'd also imagine that this rule's enforcement would be relaxed when, for
instance, a player sets up a phone to record the game and has no access to
that phone during the game, since any information the phone is recording (and
any resulting processing for cheating) would be inaccessible to the user
during the game.

------
thom
This is cool. One thing I keep hoping either exists now or will soon is the
ability to take a photo of an actual board position and get a FEN string out
(or just directly send it to Stockfish and get an evaluation). It'd be really
helpful to quickly analyze positions when I'm trying to help my kids out, cos
we don't generally keep notation and I'm not always confident that my advice
on a position is blunder-free.

~~~
indy7500
I think you'll be pleasantly surprised, this is a thing!! I thought about
making it but quickly decided computer vision wasn't up to the task yet. It
seems I was horribly wrong: [https://github.com/maciejczyzewski/neural-
chessboard](https://github.com/maciejczyzewski/neural-chessboard) Haven't
tried it myself but it looks amazing

~~~
thom
Oh man, I had just blithely assumed nobody had done it and never googled,
thanks for this!

------
heinrichhartman
This is great! Sadly it does not seem to be open-source.

I have wanted something like this for other data entry tasks like: tracking
expenses or tracking body weight. I enjoy working with pen and paper much more
than typing on a Phone or PC. Just give me a template I can print, let me
write the data to it, and have it scanned.

Is anyone aware of a product (perferably open-source) that lets me do that?

~~~
indy7500
I think I'd be willing to make it open source? I'll talk to a person that
worked with me on it.

~~~
Kratisto
I'm not sure if you posted it anywhere, but would you mind sharing what
technology you used for this? I would be very interested in contributing if it
became open source.

~~~
indy7500
[https://github.com/Messier-16/Reine](https://github.com/Messier-16/Reine)

------
gpm
> Do not indicate checks, mates or promotions at all, i.e. leave out the "+,"
> "#," or "=Q".

Some of your screenshots include the +, is this accurate?

If you don't include the =Q you can't encode all games?

~~~
indy7500
I took the screenshot with the "+" before I realized it'd be easier for now to
ignore the "+". This is my fault for being lazy, I'll change it. I looked at a
database of lichess games (40,000) to convince myself that non-Queen
promotions were worth ignoring for the beta. If promoting to a different piece
is clearly the better move, I could run a Stockfish check anytime a promotion
move is made to assume the player chose the best non-Queen promotion (since
otherwise you'd just promote to a Queen in a serious game)

~~~
gpm
Makes sense, congrats on shipping!

I suppose it's unlikely to happen in a game serious/slow enough that people
are bothering to record it on paper and upload it, but I've tricked someone
into stalemating me by promoting to a queen before. Long term I don't think
assuming people made the best move is the right decision.

~~~
indy7500
Thanks! Definitely, the check and mate things should also be fixed long term.
The nice thing is, I was able to put in tons of constraints to improve
processing (for example the last two boxes must be a square like e4) if I
ignore checks.

------
indy7500
We have decided to make it open source! Will publish in another article
perhaps, when it has enough comments added in the next two days!

~~~
mk_chan
That's great! Maybe try to integrate it with a popular library like python-
chess ([https://github.com/niklasf/python-
chess](https://github.com/niklasf/python-chess))?

~~~
indy7500
We did use that for the postprocessing actually, the part where moves are
cross-checked for lower probability predictions.

------
jamesonthecrow
Awesome job! If anyone else is working on a project like this or is interested
in learning more about applied machine learning we've got a helpful Slack
community over at Heartbeat
([https://bit.ly/heartbeatslack](https://bit.ly/heartbeatslack))

~~~
indy7500
I just posted this project on #share over on that Slack. Their #help channel
helped me a ton (especially Jameson and hart!) Fully endorse this

------
zzo38computer
Another way to make them to scan would be that there is one space you can just
write free text and then next to them you can fill in the circles to indicate
rank moved from, file moved from, rank moved to, file moved to, and promotion
(if applicable). You can indicate castling by the king's move.

~~~
indy7500
That would be an awful lot of bubbles per move, no?

~~~
zzo38computer
Yes, although there are ways to reduce it (perhaps in a way similar to
POSTNET; this would reduce the space taken up but require more fill in at
once). But you do not have to fill it in right away you can do so after the
game is finish

~~~
indy7500
Would you mind quickly explaining what you guys mean? Sorry, I'd love to
improve it but I'm not sure I get it. If you had a bunch of bubbles for each
square and piece etc. wouldn't you be taking up much more space than 5 boxes?
And it's also a huge notation change for everyone

------
indy7500
P.S. Here's a chess.com article with a video demo (if you don't want to try
yourself) and tons of info about the story / how it works. Anyone that has
advice on the website design or the software, please help! I'm new.

~~~
sp332
Looks like you forgot your link?

~~~
indy7500
[https://www.chess.com/blog/ReineChess/scannable-
scoresheets-...](https://www.chess.com/blog/ReineChess/scannable-scoresheets-
free)

------
V-2
Nice idea, but doesn't the fact that each letter / digit has to be in a
separate box (if I understand this right) defeat the purpose?

Normally at tournaments they would hand you a score sheet that doesn't have
this crossword-like layout.

~~~
indy7500
Hmm... I see why it's a hassle. How does it defeat he purpose though?

~~~
gmiller123456
The USCF rules state that the TD can require players to use a specific
scoresheet. It's not generally used in most casual tournaments, but almost
mandatory for tournaments that matter.

~~~
V-2
I don't know about the US, but eg. scoresheets with carbon coy were handed out
in the last tournament I played (Italy).

------
audiometry
I'm surprised that handwriting recognition is such a problem for you. Wonder
if you can improve accuracy by customizing the training for each user as you
grow a larger corpus of examples in each users' hand?

~~~
indy7500
Yes, that is the plan. That should heavily increase accuracy. Do you mean
handwriting recognition as opposed to character recognition? It wasn't that we
tried it and decided it wouldn't work, I just couldn't find previous instances
of accurate enough recognition with handwriting. Accuracy was my priority. My
sense was also that people would be turned off if they had to still manually
input some games, and I had no idea how many manually inputted games it would
take to reach good enough accuracy. This was when I hadn't come up with
constraining characters and such, but now I can see that (with some attempts
at handwriting recognition) that may be possible / is the next step after an
app.

It is open sourced now if you or anyone would like to contribute.
[https://github.com/Messier-16/Reine/](https://github.com/Messier-16/Reine/)

------
indy7500
Now open sourced.
[https://github.com/Messier-16/Reine/](https://github.com/Messier-16/Reine/)

------
diminoten
Pretty cool, will definitely save me some time, though I guess I used the time
converting it by hand to think through some of the moves.

------
beefman
Or you could use ChessNoteR

[https://www.chessnoter.com/](https://www.chessnoter.com/)

------
User23
This is very cool. Analyzing your own games is the single best way to improve
your play.

------
samirm
are chess players required to write down their moves?

or could the event organizers set up some overhead cameras

or make a smart board

and automate the entire thing that way?

~~~
samcal
The answer is yes to all 3! There are cameras and smart boards at the highest
level, but chess players are still required to write down their moves. Archaic
for sure, but tradition is strong with these old games.

------
petarb
That's really cool. Thanks for sharing.

------
peteypao
Oooh! Finally!!

