Their results show that they are only just barely stronger than Stockfish 8, but Stockfish 9 and 10 are stronger than 8 as well.
EDIT: Also meant to include a shout-out to http://www.lczero.org/ which is an open source implementation of AlphaZero chess. Here is their forum post for this paper: https://groups.google.com/forum/#!topic/lczero/TfmaNHI99gk
SECOND EDIT: I was wrong! They did play against a newer SF than 8, specifically, SF at this commit: https://github.com/official-stockfish/Stockfish/commit/b508f... , which was about 2 weeks before SF 9 was released, so maybe it is close in strength to SF 9.
AFAICT based on dates, they chose nearly the latest version they could have.
Recent Stockfishes recommend many of the quiet, strategic moves that he seems particularly enamoured with.
IMHO as a competitive scholastic chess player (former national U16 champion and top 3 world U10) and software engineer, it would significantly increase credibility of results. Not to mention would be fascinating to see the “ugly” games in addition to the ones handpicked by your team.
What would you do with 1000 games that's not possible with 100?
> The amount of training the network needs depends on the style and complexity of the game, taking approximately 9 hours for chess, 12 hours for shogi, and 13 days for Go.
How much would that much computing power would cost on something like AWS? That's a lot of hardware, but if you're only renting it for 9 hours... the beefiest EC2+GPU instance Amazon has currently is p3.16xlarge, which has 8 Tesla V100 GPUs, and 64 (virtual) CPUs, for $25/hour on-demand. My understanding is that a V100 is slightly more powerful than a Titan V, so does that mean you could run the Chess training (at least the AlphaZero side) for $225? That seems impossible?
EDIT: pacala below pointed out that the hardware listed was just for running AlphaZero against Stockfish, not for training it. Digging through the preprint itself, they say that for training they used:
> During training only, 5,000 first-generation tensor processing units (TPUs) (19) were used to generate self-play games, and 16 second-generation TPUs were used to train the neural networks.
So that would be... a lot more.
And the 9 hours was for training, but I don't think the article linked says on what.
Subsequent games are expensive because scaling becomes an issue
AlphaZero was a great concept and execution, but if we have to judge its relative strength, it should compete fairly. 4 TPUs (~ 4 Titan V) + 44 cores for AlphaZero vs only 44 cores for Stockfish pre-9 may or may not have put Stockfish at a disadvantage.
BTW, current, presumably balanced, TCEC 14 configurations are:
CPUs: 2 x Intel Xeon E5 2699 v4 @ 2.8 GHz,
Cores: 44 physical,
RAM: 64 GB DDR4 ECC
GPUs: 1 x 2080 ti + 1 x 2080,
CPU: Quad Core i5 2600k,
RAM: 16GB DDR3-2133
TCEC GPU server looks more modest than what A0 authors used to "beat" SF.
That reason alone, if I were Deepmind, I would not be included in those competitions. It would be horrible press for them, that would involve a ton of human error out of their control.
BTW, a match between StockFish 10 and LeelaChessZero, an open source implementation of the same idea, will be organized in a couple of days. From the LC0 blog:
Lichess.org will host a match between the mighty Stockfish 10 and Leela. It will be a 6 games match with time control of 5'+2" with ChessNetwork commentary.
Games will be played on 15th December at 17:00 UTC.
Stockfish 10 will run on 64 cores 2.3GHz Xeon, while Leela will use the latest v19.1 Lc0 with 11248 network and will run on one GTX 1080 Ti + one RTX 2080 GPU.
Will you be answering questions?
"Traditional chess engines – including the world computer chess champion Stockfish and IBM’s ground-breaking Deep Blue – rely on thousands of rules and heuristics handcrafted by strong human players that try to account for every eventuality in a game."
Here's one random test which showed improvement: https://github.com/Vizvezdenec/Stockfish/compare/5c2fbcd...5...
Bitboard b1 = double_pawn_attacks_bb<Them>(nonPawnEnemies) & b & attackedBy[Us][PAWN];
score += make_score(90, 72) * popcount(b1);
In the current computer chess championships, there are Monte Carlo engines that use the search search strategy as AlphaZero with hand crafted heuristics, and they're doing ok. But they're not as strong as AZ, which learnt, by itself, what a good chess position looks like, starting from random play
As a Go player, is there any way I could download and review the game records described in this paper?
Where as the paper describes a thousand (or more) games played between Alpha Go Zero and AlphaZero.
That's why I've specifically asked the author. Perhaps if Inufu sees there's interest, more game records could be released.
Even if (and here I agree with you) chances seem slim.
I'm excited about their work but it seems that it would be much better for everyone if they just released their work openly.
As someone said, it was in peer review for a year.
That means it could not have been compared against stockfish 9 or 10 - they were not released yet.
As someone above points out, they used in-development stockfish versions as well (2 weeks before sf9 was released), and from what I can tell, they used the newest version they could have.
If you have some good data that means you can make this statement, can you please cite it?
Otherwise, can you please not make claims of bad faith?
That said, I've checked out a few old versions of Stockfish (including the exact commit they used) and analysed the games in Table S6 in the paper. Stockfish still spots multiple blunders in its own play. Obviously these things aren't entirely deterministic, but it seems unlikely there was time trouble.
And again, just to be clear, there's little doubt in my mind that AlphaZero's evaluation of any given position is better than Stockfish's or anyone else's. I'm not even saying it can't reliably beat Stockfish. I just find it sad that the evidence of its overall strength continues to be wobbly.
If you really feel like their original setup handicapped stockfish, it seems like the best way to know would be to have that setup play your preferred setup and see what the difference is.
He says he's analyzed the published games, and found that Stockfish finds obvious errors in the way that Stockfish was said to be playing. Unless he's consciously lying about this (I'd guess unlikely?) it's not libel. And if it's true that a correctly configured Stockfish doesn't play the way it was said to play, this would be a strong indication that either intentionally or accidentally, it was not a fair playing field.
Anyway, if your point was that it might be more productive to be more polite, sure! But one might say the same about accusing someone of slander.
Please stop applying your intillenge to AI.
Edit: substantive counter-arguments would be highly appreciated
The solution to that problem is not to stop working on AI, it is to rethink how capital and power is shared among the society. Indeed advanced AI and automation may well be our way out of the exploitation of man by man.
My concern is with superintelligence. A hostile superintelligence would probably end humanity, or at least the part of it that I care about.
Basically I share Elon Musk's view on this. I think superintelligence is quite likely within a few decades or at least a century or two, and if/when it happens, our only hope is that is imbued with values we share.
Ideally, the first organizations to discover how to do this should be open or under the control of democratic governments, or secondarily, under the control of corporations that can be regulated by such governments.
The only solution to this problem is the banishment of AI. There is no other way to preserve life as we know it. AI might not provoke these changes within my lifetime. But people are very happy to protest and march for global warming even though it also will not end the world within our lifetime. There is a strange cognitive dissonance there. The logic is very similar: even if there is a small chance that it could end the world, better to err on the side of caution. The consequences of AI will be indescribably worse for humans than global warming, so why not exercise caution?
Is it a case that 'the enemy' is saying Global Warming isn't real, so there is a push to reply that not only is it real, but even worse than we expect? A concious hyperbole so people will pay more attention to a real problem you perceive as neglected? An actual belief that it will genuinely end the world because you've heard it will be very bad so you conflate it with being the worst thing possible?
I don't consider this a controversial statement. From my own reading of the relevant studies, I think there is definitely a non-negligible probability that runaway effects may make Earth very harsh or even completely unsuitable for human life, eventually. Are you saying the current studies outrule this scenario entirely or with near certainty?
I think the biggest problem with your argument is its lack of originality. Every labor-saving innovation in history has been greeted with "This is a death sentence for humanity" or equivalent sentiments, and the naysayers have never once been correct. You need to flesh out your argument by explaining exactly how It's Different This Time.
Your purpose as a human being is not to do a machine's job poorly. Think more of yourself and your fellow humans.
Once a job or task is automated, it can never be done by humans again. This is because within any market, the most competitive entity propagates and the others do not. The less competitive entities are starved of resources and stop existing. As AI approaches sentience, it will become the most competitive entity in more and more cases. When AI becomes comparable to humans, it will displace humans in every instance unless there is a conscious effort to subvert the market economy model and intentionally use less competitive options, otherwise known as abstaining from AI. However, some entities don’t care about a consensus to abstain and will use AI anyway. A country for example will always win against its adversaries if it uses AI. At the end of the day, AI will ratchet forward uncontrollably and the only entities left standing will be those who use AI. This is an inescapable and inherit characteristic of AI. Most AI experts concede to this, but wave their hands and say it’s a long way off.
The only way for humanity to continue is for AI to be banished. And by the way, this is all assuming that AI never maliciously targets human life on its own or otherwise for any reason whatsoever.
(Shrug) At least I have historical precedent on my side, so I don't have to fall back on a list of bare-assertion fallacies.
Maybe you disagree that the most competitive entity always propagates in a free market? Maybe you disagree that the world and everything in it is a kind of free market? Maybe you disagree that machines can recreate any aspect of the human mind? I literally just don’t understand where I’ve gone wrong. Please point out where I’ve gone wrong in this.