
The Steely, Headless King of Texas Hold ’Em - spot
http://www.nytimes.com/2013/09/08/magazine/poker-computer.html?hp&_r=0&pagewanted=all
======
ianstallings
The article says the machine can't be beaten. And then points out that a pro
has consistently won against it.

This article is filled with bold claims by people that want to sell the idea
IMHO. I'm not buying it because even limit texas Hold'Em has never been
_solved_ mathematically by super computers, let alone a single machine. Limit
Hold'Em is close to being solved but if you check out the last match of pros
against a supercomputer you'll find that it's close, but the computer is not a
clear winner: [http://poker.cs.ualberta.ca/man-
machine/Competitors/](http://poker.cs.ualberta.ca/man-machine/Competitors/)

I'm just not buying it. Can it beat the average player? Probably. Pros do it
everyday. But can it beat a pro consistently and claim a clear victory over
humans? _To be determined_.

Edit: I found a take from High Stakes Limit Pro Anthony Rivera on 2p2:
[http://forumserver.twoplustwo.com/showpost.php?p=29762774&po...](http://forumserver.twoplustwo.com/showpost.php?p=29762774&postcount=28)

His stance is that it's probably a break-even situation where you are
basically playing another pro, and that he'd rather be playing craps. I guess
they really have made a great machine. :D

~~~
Someone
Is it actually possible to mathematically solve those games? Suppose player
Alice bets $100; how does player Bob judge that bet? As a strong hand or as a
bluff? Any software program will have to model what's inside the head of its
opponent, and we know way too little of that to do that.

Moreover, if we could, the opponent may choose a different strategy. It may be
like a game with nontransitive dice: if you figure out which dice I use, you
can beat me, but if I figure out that you figured that out, I can pick
different dice and beat you
([http://en.wikipedia.org/wiki/Nontransitive_dice](http://en.wikipedia.org/wiki/Nontransitive_dice))

I think it is possible to write software that statistically beats any
opponent, but proving that it does is way harder.

~~~
jib
Yes it is possible to mathematically solve limit poker (as well as no-limit
poker, but no-limit poker is orders of magnitude more difficult - they are
effectively different games).

The definition of solvable is "Does there exist a strategy that, regardless of
opponents play, is not a losing strategy" though (because the game is
symmetrical). It is not possible to solve for "Does there exist a strategy
that, regardless of opponents play, guarantees maximum profits". You can solve
it for not losing money, but you cant solve it for making the max amount of
money.

Trivially it's not possible to write software that beats any opponent (because
what would it do playing against itself?) . Less trivially, any game that has
a finite amount of decisions (and limit hold-em does) has at least one Nash
equilibrium, so there exists a strategy that will at least have you break
even.

The way to solve the game is to calculate your odds of winning based on
previous actions and ensure that you take actions that make any future
decision of the opponent have the same outcome (to reach a Nash equilibrium).

That strategy hasnt been calculated yet, but the best limit players are most
likely playing very close to it, at least if you compare to the best no-limit
players playing no-limit (the variable amounts possible to bid in no-limit
multiplies the possible strategies massively).

------
scotch_drinker
In his book Super System, Doyle Brunson wrote that a computer would never be
able to play elite poker because you played the man, not the cards or the
game. He greatly underestimated the power of modern computing it appears.

Personally, I'd rather see resources go into a robot that can cook for me but
development follows the money I suppose.

~~~
menubar
..or, he greatly underestimated the fact that the computer knows what cards it
has dealt to you.

Isn't this obvious?

~~~
gonvaled
Well, exactly! I have three aces, and the computer has double pairs. We are in
the turn. I go all in!!!

The computer calls ... and deals itself a full house. I am bankrupt :( How do
I know? How can I ever trust playing poker with a machine, when it is doing
the dealing?

Or, let me put it this way: I will play any machine, no limit, if I can do the
dealing (secretly, that is, as the machine is doing).

Another issue entirely is chess: no secret dealing going on. All are playing
with the same in-game information.

~~~
Permit
This is easily remedied by retaining a human dealer. It's fairly trivial to
create a computer that can identify dealt cards on the table.

~~~
gonvaled
That would be fair, to be sure - or, to be precise, as fair as playing any
other table in the casino. The dealer can still be cheating, but doing that in
the open does not raise any further suspicions as any other table in the
casino would raise.

------
swang
1\. Is there anything Phil Hellmuth won't shill for?

2\. When this machine showed up ~2? years ago, there was a thread on 2+2 about
it, and it would sometimes do some weird things like not value bet in obvious
situations. They explain it in this article by saying it's, "playing dumb" but
that seems like it would be a huge leak against Limit Hold'em HU specialists.
I am guessing that they assume that they can make up for it in the weaker
players losing consistently against this machine.

3\. It is kind of annoying that the guy is proud that he "broke" a 24 year old
player.

------
vladimirralev
It's probably carefully tuned to calculate the raw probabilities and it also
forces the player to reveal his cards to run statistics on his style. That's
why it's the limit version. This machine has no chance in a classic no-limit
texas where the opponent may be only profiting from large bets unjustifiable
in expected value most of the time with very little style statistics.

~~~
bvk
The game can't run statistics on the player's style.

>Casino commissions, however, mandate that a gaming machine cannot change its
playing style in response to particular opponents.

The algorithm is encoded in a neural net; there is no calculation of raw
probabilities involved, at least in the sense programmers are used to.

~~~
vladimirralev
Interesting. I doubt they would design a neural net without the probability as
direct input though. I don't think the machine can be this good especially
with such limitations. It may be exploiting some average-gambler trait and
ultimately will be beaten once the pro players figure it out.

------
tom_b
Quite fun to see neural networks in the news. A million years ago I took a
class on neural networks, using backpropagation to train networks to do image
classification.

I will never forget the feeling of "you've got to be kidding me" when the code
I had written was able to successfully classify a huge percentage of the
validation image set. I really should head back in that direction and get with
the machine learning.

Re: poker software. Before the feds crushed online poker, I had already given
up on low-limit limit hold-em. It had clearly become a bot and augmented-
player race. I would assume given game theory and Bill Chen's book that some
enterprising hackers would encode pretty decent no-limit bots as well. If any
hackers know anything about successful no-limit bots, I'd enjoy reading about
it.

------
reillyse
I wrote a break-even no limit cash poker bot when I was in University. So I've
spent a lot of time researching and writing code in this are. Limit poker is
fairly well solved by bots at this stage. This is because the risk is limited
on each hand and the grinding nature of the computer (i.e. it's ability to do
the best thing for the longest amount of time) makes it a winner. Playing no-
limit poker, it's a totally different scenario and a real challenge which is
why I enjoyed it so much. Any slip up can cost you your entire stack. By slip
up I don't mean a bug or clicking the wrong button, I mean a flaw in your
strategy. Maybe something as simple as betting 4 big blinds instead of 3 big
blinds pre-flop against a raise from a player will end up after a couple of
rounds of betting of being played for your entire stack, instead of being
something you could fold on the turn. Anyway, long story short. Limit poker is
very solvable just by enumerating the various possible outcomes. No-limit cash
poker is a different beast, it's exponentially harder than limit poker. It'll
be a long time before somebody solves no-limit cash games.

------
b0b0b0b
Assuming the machine is stateless, why can't the space of hole cards be mapped
out and and strategies/responses enumerated?

But ultimately, the whole enterprise just seems distasteful: "Look, there was
a 24-year-old who had beaten it for a while. Now he’s broke. And I think this
machine had something to do with his demise."

~~~
iopq
There's 1352 card combinations, many more flops, many more turns, many more
rivers and also 1352 combinations for your opponent too and 3 actions per
round of betting

There's more poker situations than amount of atoms in the universe

------
snake_plissken
I don't understand how the machine can change the style of its play if:
"Casino commissions, however, mandate that a gaming machine cannot change its
playing style in response to particular opponents."

Yet the article states that the machine has multiple personalities (passive,
aggressive, etc) and will sometimes throw hands.

~~~
pessimizer
Changing strategies randomly or in response to opponent play is probably not
what's being referred to here. What would be banned would be the machine
noticing that John Doe hasn't won in a while and his play has slowed down, so
lets throw a few hands to keep him interested.

In other words, machines are allowed to shark $player, but not allowed to
target John Doe specifically.

------
elliptic
So I know a little about neural networks and ML in the (simpler?) context of
image classification etc. Anyone have a good reference/introduction to using
it for this sort of work?

------
dwaltrip
Very interesting tech, but personally I think I would struggle with the moral
ramifications of working on a project that makes money by turning people into
suckers

~~~
AsymetricCom
But the applications of this technology reach beyond gambling, into finance
security and even foreign policy.

------
adam-a
Is there any guarantee that the machine isn't simply reading your hand and
then occasionally letting you win to keep the payout at ~80%?

~~~
ubernostrum
An opponent who knows what's in your hand and bets based on that knowledge is
not terribly hard to detect. The way to make it hard to detect is to have it
not make bets based on knowledge of your hand.

------
bitwize
The illustrations accompanying this article remind me of the ones in David
Ahl's _BASIC Computer Games_.

------
hannibal5
Illegal poker bots grind constantly on low level online poker games and take
the money from low level players by just value betting them.

Computers can easily calculate probabilities and expected values for every
hand. So why they are not beating professionals?

Poker has no optimal strategy that wins against all other strategies. To play
poker in higher level you must model the strategies of others, including them
trying to model your strategy.

1\. In the lowest level it's just maximizing expected value based on hand
probabilities.

2\. In the second level you try to learn the strategy of your opponents and
maximize value against those strategies.

3\. In third level you try to figure out how much your opponents have figured
out your strategy and you change the strategy so that you maximize value when
your opponents play against what they have learned from your strategy so far.

4\. ... and so on. It' goes meta.

If you want to create ultimate pokerbot, it has to be able to model the minds
of it's opponents. it hast to be able to detect leaks in it's own game and
close them down. It must go meta all the time. It must learn how to understand
how others think and how they think about it and change it's behaviour
constantly.

~~~
dschatz
Head's Up Hold'em has a nash equilibrium. Therefore there is at least one
mixed strategy (I do X with probability P in Y situation) which cannot be
negative expected value to any other strategy. In this sense, there is an
optimal strategy. It doesn't mean that it is the maximum expected value
against a particular opponent, but no opponent can win by playing (which is
largely the goal of a casino).

Opponent modeling is purely advantageous, but not necessary

~~~
ianstallings
I found this artile by Bryce Paradis that elaborates on using a Nash
equilibrium for optimal play. He is known for bringing advanced mathematics to
the game of limit poker and winning a small fortune because of it. Here is his
take:

* Q: What’s a Nash Equilibrium or “game theory optimal” strategy? – Failed Math, Port Perry, Ontario A: An equilibrium strategy is one that wins the most money possible against a perfect opponent (this does not mean an opponent who can see your cards, but one who always knows your range whenever you take an action and makes the best choice against that range). In the game “rock, paper, scissors,” the equilibrium strategy is to randomly choose between the three options, choosing each one a third of the time in the long run. Finding equilibriums in poker is much more complicated, but the concept can be useful when you’re playing lots of hands against tough opponents. For example, if your opponent bets half the pot on the river after a particular series of actions, the pot is offering him 2-1 on his bluff. If he were a perfect player, the right thing to do would be to call his bet a third of the time, since if you called more he’d exploit you by never bluffing and if you called less he’d exploit you by always bluffing. In reality, of course, our opponents are never perfect, and so the idea of playing an equilibrium strategy at the table is usually pretty academic. *

[http://pokerpromagazine.com/proscorner/bryce-
paradis/](http://pokerpromagazine.com/proscorner/bryce-paradis/)

~~~
hannibal5
I can see the possibility for confusion here.

You can certainly use Nash equilibrium when you have figured out the
strategies your opponents are using. This is what Bryce Paradis is talking
about. It can have practical value when playing Heads Up.

But If we are talking game theory and "solving poker", there is no single
winning strategy that works against all other strategies and you can't
calculate single Nash equilibrium that would be optimal in actual game against
specific strategies.

~~~
iopq
Yes, you can.

Your opponent has 1352 different hand combinations. Assume he is playing the
Nash equilibrium strategy. Make the perfect plays based on this. If he plays
worse than the Nash equilibrium strategy, you beat him. If he plays perfectly,
you tie.

Assuming your opponent plays perfectly works in chess. Chess programs are
stronger than the best humans now.

~~~
hannibal5
>Assume he is playing the Nash equilibrium strategy.

You you can't do that assumption because you don't know what the strategy is.
You can calculate Nash equilibrium only if you know the strategy your opponent
is using. In full no limit hold em there is no single strategy winning
strategy, so you don't know the strategy your opponents are using.

~~~
jib
No you don't need to know what the opponents strategy is. You calculate based
on worst case (ie op playing perfect) and worst case is you break even. There
is no way to maximize profit but you can play unexploitable, ie at a minimum
not lose and possibly win assuming op doesn't play perfect.

~~~
hannibal5
In poker optimal strategy is not winning strategy.

An optimal strategy’s goal is to loose the least against any arbitrary
strategy. It is a strategy that is impossible to exploit in poker because
poker has antes.

Poker players must seek maximal strategy. A maximal strategy’s goal is to win
as much as possible against a specific strategy.

~~~
iopq
That's just not accurate. An optimal strategy beats everything but itself,
against which it ties. Are you talking about rake? Because antes are included
into the poker strategy.

For example, you would raise more often when the antes are higher, regardless
of the other player's strategy.

~~~
jib
I think the point he is trying to make is that if your goal is to maximise
your profits then it is not always optimal to play the nash equilibrium
strategy. That is true.

The optimal strategy beats everything but an equally good strategy, and ties
against itself, but it doesnt necessarily maximise profits against other bad
strategies.

If you are able to identify flaws in your opponents strategy then you can play
non-game theory optimal to increase your profits against that perceived
strategy. Doing so comes at the cost of you yourself no longer playing the
best strategy though.

There exists strategies that gives higher yields vs certain unbalanced
strategies than the game theory optimal strategy (or strategies - for all we
know there are several optimal strategies in limit hold'em).

For instance - in limit poker if your opponent will never raise, call every
street, but not call the river with anything less than a pair, regardless of
what you do, then bluffing every river is a more winning strategy than the
game theory optimal strategy. The game theory optimal strategy would include
times when you do not bet the river, for balance, but knowledge of your
opponent's flawed strategy would tell you that betting 100% of the time has a
higher yield.

~~~
iopq
This is true, but the program can't adapt due to the regulations, so it tries
to play a Nash equilibrium strategy.

