
Evolution of chess: Popularity of openings over time - rhiever
http://www.randalolson.com/2014/05/26/a-data-driven-exploration-of-the-evolution-of-chess-popularity-of-openings/
======
dfan
The Pirc spike in the 1850s looks interesting visually, but keep in mind that
your data set back then is incredibly small. Looking at my own database, the
spike seems to be entirely due to some guy name Mahescandra playing it 57
times against Cochrane. It certainly doesn't have anything to do with the
increasing popularity of 1.d4 40 years later.

I've never heard of Mahescandra, but Cochrane is the guy the famous Cochrane
gambit in the Petroff is named after, where White sacrifices a piece on move 4
(1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nxf7).

~~~
rhiever
I double-checked my data set to look into this, and you're right. Nearly all
of the Pircs in that time period were done by a guy named Moheschunder
Bannerjee:
[https://en.wikipedia.org/wiki/Moheschunder_Bannerjee](https://en.wikipedia.org/wiki/Moheschunder_Bannerjee)

He contributed to the development of the Indian Defence.

~~~
Blahah
Why do you all have your own datasets of historical chess moves? Is this a
popular playground for data analysis, or is it something chess learners study?

~~~
TylerE
Chess compresses very well. You can fit pretty much every meaningful game ever
played into 10gb or so.

------
cven714
Chess opening trends are like fashion--some high profile player(s), always on
the search for new ideas, finds a resource in an unpopular line and suddenly
it's all the rage. Everyone is playing it, working out the complications,
finding ways to defend or neutralize the lines, then interest wanes until
someone uncovers a fresh new plan somewhere else and the cycle repeats. Other
times though, new resources aren't found and a line mostly dies out, like the
Kings Gambit.

So what I would be interested to see from your data set is a relation between
opening performance and its popularity. Did people stop playing the Pirc due
to sub-par results compared to other openings at the time (like I imagine
happened with the Vienna) or did it simply fall out of fashion? It would be
interesting to know which lines always had good results, but just stopped
being popular for whatever reason. They could be due for a revival.

~~~
dfan
Unfortunately, it isn't always obvious from the data which of your
interpretations is correct. Take the following two fictional cases:

Case 1: The Limburger Gambit was popular for a few decades, but then
grandmasters started playing the Limburger Gambit Deferred instead, so it fell
out of fashion, although it is still a perfectly good opening.

Case 2: The Limburger Gambit was popular for a few decades, but then a
response was found that refuted the whole line. It was played in one famous
game, after which everybody abandoned it.

As far as the data set is concerned, the difference between these two cases is
a single game, so it's not going to show up in the stats.

~~~
cven714
This is true, but I think there are many more case 2s where rather than an
entire line being refuted in a single blow, gradually one side was able to
find antidotes and the line stopped being a good practical choice. The Vienna
is an example. So while it isn't always obvious, I'm hoping enough fall
between your two Limburgers to be interesting.

~~~
dfan
The two biggest examples I can think of of "opening fell out of fashion but
then was brought back at the highest levels" are Kasparov reviving the Scotch
in the 1990s and Kramnik reviving the Berlin variation of the Ruy Lopez in his
world title match with Kasparov in 2000.

------
themodelplumber
As a novice player, it's frustrating to read stories about players beating
other players "by the book," i.e. they know more openings than the other
player does, so they play some unexpected opening and just destroy the
opponent out of the gate. I guess this is possible in just about any game, not
just chess, but it seems to reduce the optimal learning process for a novice
to "start memorizing things". Bah. Edit: Not that memorization is bad or
boring by necessity, but finding a way to make the memorization fun and
interesting is in itself frustrating, knowing that you could learn openings
really fast just by brute-forcing the various lines into your brain with
things like memory palace techniques.

~~~
danielbarla
This is an argument that is raised a lot in chess circles. While there's some
truth behind it, it tends to be applicable much more in an environment where
both players are trying really hard to play "in book".

Before you get too deep into openings, you should have a solid understanding
of the strengths of individual pieces. This leads to a better understanding of
why you'd want to put your pieces in certain places. It kind of explains the
broad strokes of openings, without the little traps here and there. This is
known as the Russian school of teaching.

Basically, learning from this direction, you can spend much less time studying
openings. You might want to know about few common traps in common situations
that you tend to use, but that's not the focus. And you'd be surprised at how
well this works - barring a tiny number of games where you're caught out, your
better understanding of the overall game will win out.

Essentially, overspecialising in opening repertoire at the novice level is a
kind of arms race which only really works against other similar players. It's
not a good long-run strategy.

------
adamconroy
The analysis is interesting. However I'm not sure it has much practical value
due to transpositions. For example, as white I play 1.Nf3 and if black plays
d5 I play d4 and we have a d4 opening. If black plays c5 I play c4 and
depending on what black does it will transpose into either an English opening
(1. c4) or a maroczy sicilian (1. e4) or an indian defence (1. d4).

So basically, my opening move would be classed as 'other' but really it is one
of 1.d4, 1.e4, 1.c4 in terms of the classifications of this post.

~~~
rhiever
If any of the alternative paths were common enough, they would show up in the
charts as well. I didn't limit the analysis to a particular set of moves; I
simply counted all of the paths present in the data set and showed the most
common ones. This is why two variations of the Indian Defence show up in the
"White's second move" chart.

I think it'd be interesting to try to combine all possible paths for an
opening into a single count, but that would probably be complicated if
multiple openings can be reached through the same path. (e.g., which opening
would the shared path be assigned to?)

~~~
dfan
The traditional way to handle this is by classifying a game according to the
last cataloged position that occurs in it. This is how ECO classification
works; you can see its catalog of positions at
[http://www.chessgames.com/chessecohelp.html](http://www.chessgames.com/chessecohelp.html).

For example, just after White plays 1.Nf3, the game is classified as A04, but
after 1...d5 2.d4, it's now officially a D02, over in the Queen's Pawn
category, just as it would have been if the game had started 1.d4 d5 2.Nf3.

Databases usually keep track of chess openings played by ECO code rather than
by specific moves, exactly so that these transpositions are handled smoothly.

~~~
rhiever
I wish I knew about this earlier! Thanks for explaining it to me though. :-)

------
geebee
I noticed that the queen's pawn opening has grown in popularity considerably.

I don't really play chess anymore, but two decades ago, I played quite a bit.
Much of the time, it was against a computer. I couldn't win. I finally found a
book that really worked for me, and it taught me the "stonewall" approach,
where you (playing white) advance the queen's pawn, lock up the game, and
start maneuvering very tactically to restrict the movement of black's king and
get a mate.

I found that this was the only way I could beat the computer - largely because
it allowed me to think along a very few narrow lines (often involving a
sacrifice), while the computer would get bogged down in useless searches. This
was all in the late 90s, my guess is that it wouldn't work anymore.

So, I'm wondering if the surge in queen's pawn opening is the result of people
being conditioned to play against computers? And it brings up another point -
I've _heard_ (man, I wish I could find a link to this theory on the web) that
chess masters who lament the rise of computers actually aren't particularly
bummed that computers can beat the strongest human chess players. They're
bummed that the best human chess players are now playing a style that is
heavily influenced by years of practice against a very specific style of
opponent - a powerful computer that can't really make a "mistake". Young
people have opportunities to train that never existed before, but because they
don't train against stronger _human_ players as often, some of the style of
the game has been lost. I'm not (and never was) good enough at chess to know
if this is true, but if anyone has an angle on it, please chime in!

Oh - by the way, a very satisfying ending here. I knew a very brilliant guy
who I couldn't beat at chess in high school. After I read this "stonewall"
book, I was all ready for him. I advanced queen's pawn, all read to lock it
down and take away his advantage. He simply advanced his king's pawn,
basically telling me, go ahead and take it. The wide open game against an
inferior opponent was worth far more to him than a pawn.

Good for him, though not so good for me ;) That's the kind of thing you do
when you're experienced playing against people rather than computers.

~~~
dfan
Was that book with the Stonewall Attack "How to Think Ahead in Chess"?
[http://www.amazon.com/How-Think-Ahead-Chess-
Techniques/dp/06...](http://www.amazon.com/How-Think-Ahead-Chess-
Techniques/dp/0671211382)

I don't think that the current popularity of 1.d4 is due to playing against
computers. The main reason it is currently more popular than 1.e4 at the
highest levels is that Black has found a very effective way to counter 1.e4
(the Berlin variation of the Ruy Lopez) that makes it difficult for White to
achieve more than a draw. This high-level popularity then trickles down to
lower levels.

It is definitely true that the style of play has changed somewhat due to
computer engines, but it's not so much the result of playing against computers
(top players generally don't find that very rewarding), but of analyzing with
computers. You can play through a game or an opening and explore variations,
with the computer constantly telling you what it thinks the best moves are and
who is winning by how much. As a result, players become trained to evaluate
positions closer to the way a computer would. One example is that computers
don't mind grabbing material and defending an unpleasant position for a long
time if they don't think that the opponent can break through. This has given
players confidence to be more materialistic than they were in the past.

~~~
geebee
Yes, that was the book!

This also reminds me of a Martin Amos novel "the information" (based on the
publication date, I'd guess I read it around the same time I was learning the
stonewall attack). There's a section where one writer is determined to defeat
his friend at everything (tennis, chess, a few others), and he hires people to
help him.

The chess teacher, I believe, teaches him something that sounds like the
stonewall defense, and reflects that it almost feels like he's teaching
someone to _cheat_ at chess rather than play it. The writer isn't cheating of
course (at all), but I think it feels this way to the chess master/instructor
because he's learning the stonewall because he doesn't really want to engage
in the actual competition. He wants to sidestep it in order to win.

I'm not trying to knock the queen's pawn defense, more the way I (and this
writer) were trying to use it. I'm sure queen's pawn openings can be very
creative. But in my case, this is why I considered my buddy's play of the
kind's pawn a "satisfying ending." I was (to use the tennis analogy) using a
pusher's approach. When you watch two people playing a game, and it's clear
that one person is trying to win by shutting down the game rather than really
playing it, you tend to root for the person going for creative play.

------
Cogito
The visualisation I would most like to see with respect to openings is a 'heat
map' like diagram, showing how likely a given piece is to occupy each square
after _n_ moves.

Are there any squares never used after the first move?

Just what weight is given to the centre of the board after the first few
moves?

This sort of information could be communicated really well with a view of the
board, and the pieces expected to be in each square.

------
bane
Out of curiosity, is chess solvable yet by computers? Meaning, is it possible
to simply brute force every possible legal game up to _n_ moves and determine
all the winning and losing move sets? What's this number look like
theoretically?

(I'm sure in the general sense games with a very large n aren't as I suppose a
game could be played in perpetuity)

~~~
patrickmay
No, it is not yet possible to brute force all chess games. The largest extant
endgame database has full solutions for seven pieces on the board. Once you
get to that point, chess is solved. ;-)

~~~
rmrfrmrf
Just to clarify what you alluded to, it's theoretically possible for chess to
be solved, and progress is being made slowly, but surely (it took around 7
years for chess to be solved up to 7 pieces). Of course, with each additional
piece comes an exponentially larger set of positions, so progress from 7 to 8
pieces should take much longer assuming there aren't any massive breakthroughs
in computing speed.

------
Matetricks
Great blog post. It's interesting to compare my knowledge of opening evolution
with the historic data. During the Romantic Era, King Pawn games were clearly
the norm. When Reti and Nimzowitch introduced hypermodernism in the 1920s
Indian openings became much more popular.

------
alien3d
i don't know much about opening but i play chess frequent in my younger age
and sometimes computer. It's just a pattern.But to win with computer,must
follow non pattern/opening to win.. Very long time not playing chessmaster
software.

------
kristopolous
I'm under the impression that introducing the Queen too early (say < 6 moves)
usually leads to it being taken and that player losing.

I don't know how much this is actually true though. That would also be a good
thing to look over

~~~
dfan
Indeed, one possible consequence of developing the queen early is that it can
be harried with tempo by the opponent; that is, he gets to make you respond
while making developing moves he wanted to make anyway.

However, except in cases of gross incompetence it doesn't lead to your queen
being taken, it's just suboptimal. There are also plenty of exceptions to this
rule, of course. The most obvious one is the main line of the Scandinavian
Defense, 1.e4 d5 2.exd5 Qxd5. This isn't a grade-A opening but it's a solid B+
and occasionally gets used at the highest levels.

------
Anon84
Where does one download the data? The first post in the series links to the
website but there doesn't seem to be any download links anywhere

