Poker is a Skill Game, but Cigital Study Flawed

Created By:
29 April 2009
Posted in: The Poker Reporter Blog, Cold Hard Facts
Cards and chips

There has been quite a flurry of reporting in the poker community and elsewhere (the Wall Street Journal had coverage) of the findings of a study by Cigital, a consulting firm in Washington, DC.

The report presents what some are calling "definitive" data that "proves" that poker is a game of skill.

Now, poker is a game of skill. Luck plays a role, of course, in the impact of the random turn of a card, the flukiness of the flop, the unlikely river card. That's not only the nature of the game; it's an inherent feature of every interesting thing that people do.

But the Cigital data do not support the conclusion reached. Let me explain why.

In scientific research we hold dear to a singular principle: you have to control alternative possibilities that might be causing the effects seen. The Cigital study does not do this. In fact, it has a large logical flaw in its design.

They analyzed 103 million Hold'em hands (every one played at the micro-level games offered on Poker Stars in December, 2008). They found that fully 75.7% of them never went to showdown.

In fact, in only some 12% of the hands was the pot shipped to the player who actually showed the best hand.

Given the size of the data base, these numbers are virtually certain to be accurate estimates.

Magazines, web sites (including this one) and, of course, the Poker Players Alliance (of which I am a proud card-carrying member!), greeted them with the claim that they show that the game must be one of skill.

If the cards are irrelevant fully three-quarters of the time, then, heralded one publication:

"The player could be holding two pieces of blank paper and it would make no difference."

This conclusion is not warranted.

Those players who acted sufficiently strongly to force their opponents out of hands, persuaded them to muck their cards or lay down perhaps the stronger hand, may have done so only because they started with good cards, in which case cards do matter.

Tom Dwan
LOL at micro games.

The Cigital data base came entirely from micro-level games (10¢, 25¢ and 50¢ blinds). At these levels it is rare to find players will be make big bluffs with junk or reraise with air. Most play their cards in a fairly straightforward manner.

Hence, when they win an uncontested pot, it is likely because they have a hand that they believe is the best hand right now or can easily become the best by the time all the cards have been dealt and their opponents don't.

The definitive study will not be one that tracks hands, it will track players. The really convincing data would be those that followed individuals in a variety of games over an extended period of time and found that some had consistently better results than others.

Unfortunately, it isn't going to be easy to do this, as another study of online poker by Ingo Fielder and Jan-Philipp Rock at the University of Hamburg discovered.

They examined the results of over 51,000 online players and found, to their (and my) surprise, that the majority of them play fewer than 100 hands, lose their bankroll and quit.

Importantly, what this implies is that there may be a higher percentage of winning players online than previously suspected because of the vast sea of the wildly incompetent who buy in, get crushed by the knowledgeable, go broke and vanish.

Phil Hellmuth
If there's so much skill why can't Phil win them all?

The Fielder and Rock study deserves wider distribution, which I'll give it in a future column. They conclude, if you're concerned, that poker is overwhelmingly a game of skill.

Mark Twain had it right more than a few years ago. Commenting on a dispute between a two men about whether poker was a game of luck or skill, he quipped, "Well, why not just let 'em go in the back room and play for a couple of hours. Whoever comes out with money is right."

It's important that we get this right. We are in a war, a test of will with legislators and the legal establishment to show that poker is not gambling, in the usual sense as an enterprise dominated by luck and chancy events.

We do not want to go into court or into a hearing equipped with illogical arguments that will crumble under scrutiny.

Click through to check out Carl Bialik's take on the study on his Wall Street Journal blog.

Author Bio:

Arthur Reber has been a poker player and serious handicapper of thoroughbred horses for four decades. He is the author of The New Gambler's Bible and coauthor of Gambling for Dummies. Formerly a regular columnist for Poker Pro Magazine and Fun 'N' Games magazine, he has also contributed to Card Player (with Lou Krieger), Poker Digest, Casino Player, Strictly Slots and Titan Poker. He outlined a new framework for evaluating the ethical and moral issues that emerge in gambling for an invited address to the International Conference of Gaming and Risk Taking.

Until recently he was the Broeklundian Professor of Psychology at The Graduate Center, City University of New York. Among his various visiting professorships was a Fulbright fellowship at the University of Innsbruck, Austria. Now semiretired, Reber is a visiting scholar at the University of British Columbia in Vancouver, Canada.

More articles from Arthur S. Reber:

Please fill the required fields correctly!

Error saving comment!

You need to wait 3 minutes before posting another comment.

Comment added. Thank you!

Nate Horst, Econometrician 2010-08-24 00:45:14

The author is correct. The "cards" effect is indirect and influential in determining the "decision" effect. Decisions in poker are made not with blank pieces of paper, but with actual cards. Arguing that they "could" be blank pieces of paper due to the fact that they are folded is a logical fallacy when they are in fact not blank pieces of paper. In econometrics, we call this "multicolinearity," a significant amount of which will undermine any proof of relationship with the dependent variable (in this case, "winning)." The Citi study proves nothing with regard to the relative statistical significance of chance/skill in determining winners at poker.

Patrick Fleming 2009-05-08 16:18:00

Mr. Reber, I hope you are still reading comments posted here.

I am the Litigation Support Director for the Poker Player's Alliance and in that capacity I was a major force in seeking to have the Cigital study done.

Your basic criticism of the study is not invalid, but is actually beside the point. The study does not BY ITSELF prove that poker is a game of mostly skill. What is does is validate a key and very important part of the overall argument that poker is a game of mostly skill.

What is validates is that poker is most frequently resolved by decisions made by the players, not by the turning over of the cards. And even when the cards are turned over, 50% of the time decisions of the players have thwarted the result that would have occurred had the cards alone been the deciding factor.

Proving this was extremely important to the overall effort. Most folks unfamiliar with poker assume that turning over the best hand is how poker is decided most of the time. They therefore conclude that poker is mostly decided by what cards a player is dealt.

By showing these folks that the actual deal of the cards does NOT decide the outcome the vast majority of poker hands, we then force them to consider and think about how the game is played vs. how the game is dealt.

Once we have them thinking about how the game is played, we can then educate them on all the various factors BESIDES the cards a player holds that must be considered in making the decision to check, bet, call, raise or fold.

And that is where we ultimately prove that skill at making correct decisions is only partially based on the cards and, in fact, most often the actual cards are the least important factor in making that decision.

And thats when the light bulb goes off over their heads and we have convinced them, as we have in 3 courts so far, that the evidence that poker is game of mostly skill is "overwhelming."

Patrick Fleming
PPA Litigation Support Director

Salvatore Apodaca 2009-05-06 21:29:00

I respect the data. I also respect your questioning of the data. And I believe, also being a card carrying member of the PPA, that this matter of skill versus luck cannot be answered simply. The reason for this is the elements of choice, will, endurance, deception and determination. Your examination of the study and I believe the study itself does not take into account the specific skill level of the participants and the opponents nor the amount of hours they had played continuously, whether they were simultaneously playing multiple tables or even multiple sites and if they normally played at these higher stakes, the strategies they were employing.

Taken these other factors into account is important for resolving the way a person plays certain hands, certain games, certain stakes and even on certain days. One of your photos begs the question, "If there's so much skill why cant Phil win them all?" There are a lot of answers for this. First of all that question assumes that Phil is the most skilled poker player. Let's say, "he is the most skilled player." There are other factors (most of which I mentioned above) besides luck and chance that can destroy his bids for victory. You can also make this same argument in the game of golf. If it's all skill then why can't Tiger Woods win them all? I think you can give me a great number of reasons and they would probably have parallels to the reasons Phil Hellmuth has troubles.

I suppose I am trying to say that each player takes a different mindset or psychology into each and every game he plays just as a golfer or boxer would. None of us play the same every day or every hand. Poker relies on your ability to adapt to an ever changing landscape of situations that data alone cannot predict or examine.

The data and the ensuing arguments show that the game of poker is complex and does require a good measure of skill. That is becoming more and more obvious as we progress to an inevitable "day in court".

Salvatore

Arthur S Reber 2009-05-02 19:14:00

"Spadebidder":

I had the publicly presented materials to go on and there it seemed that the lion's share of the data came from the lower levels.

However, even excluding the micro-level games, the point I was making still holds. The cards each player holds are playing a significant role in the way in which each hand is played out. This will be true no matter what level you examine ---- although the relative contribution diminishes as the stakes rise.

Of course, this doesn't negate the main point, that the skill component outweighs the chance element. It simply points out that the strong conclusions drawn from the "hands" approach taken by this study has a logical weakness.

I see the Cigital study and the "player" approach taken by Fiedler and Rock, not as conflicting, but as converging lines of evidence all pointing to the same conclusion.

See this week's column for more details.

Arthur

----------------------------------
DaveGK:

The bot experiment has a huge number of factors. A version of what you suggest has been done. If you sift through my earlier columns you'll find two on "Bots" that discuss this issue.

Arthur

spadebidder 2009-05-01 04:49:00

Mr. Reber -
Your information about the study is wrong, and it affects your line of reasoning.

The micro stakes were the levels actually excluded from the study, and the majority of the 100+ million hands were $1 blind levels and higher. They included a small portion of .50, ,25, and .10, and excluded all lower ones. The reason is that at .50 and below it plays too much like play money, as you alluded to. That's why they excluded them. I know this is not entirely clear from the published report summary section due to wording, but I've had discussions with the study author's both during and after the process.

You may want to rethink your argument.


davegk 2009-04-29 01:07:00

Why not run a study involving a series of heads up matches against an AI opponent? Find a large sampling of pros, amateurs, and complete novices to play against the computer and deal the exact same cards to each person. If poker is a game of chance, then each person should have exactly the same results at the end of a series of matches. If poker is a skill game, then the players who have proven to be "better" over time (through proven statistics of previous play) will have better results against the computer than the novices and amateurs. You can then classify each player's performance with a rating system. Run a second round of matches with the same players and see if the results line up with the assigned ratings of each player determined in the first round. Repeat for multiple rounds until satisfied with the amount of data collected.

Bloggers