I Read it in a National Enquirer Survey, it Must Be True!

Is extrapolation from forum 'game problem' posts to indicting a game as a steaming pile 'o poo the same as taking a Cosmo survey at face value?

I think so. Every day, I see Forum Fights where particularly vociferous posters scream bloody murder over their current 'problem' game, lamenting how poorly coded some part of the game is, how anserine the developers are, usually by members that have demonstrated through prior diatribes they likely couldn't tell a line of code from a line of coke.

There are of course, mixed in with the cacophonous whiners, posters with real, genuine problems. It is often stated by some in the forums that the number of complaints over a game proves that the game has problems, and not just for them, but for everybody. These posters will argue to no end that this is so, even when shown contradictory evidence such as the vast majority of players that seem to be enjoying the game without issue. To me, their claims have always been analogous to making the logical leap from seeing everyone in a hospital is ill to the population of the planet is sick.

Can we make any valid statistical inferences from what we see in a forum? I think not. I'm more on the number theory side of things, but I'm comfortable enough with my background in statistics to state that we really can't make any valid inferences on the whole population of players for a game from what we see in forums.

Basic statistics (AP High School level) reveals to the learner some facts that at first seem perhaps implausible, and that would most likely seem implausible to the lay person. That valid inferences for large populations can be made from small samples (e.g., a few hundred persons sampled from a population of say one million that is being studied), and that the larger the population under consideration the smaller the sample size needed (expressed as a percentage of the total population) do seem a bit hard to believe.

But basic statistics shows us this is so. The interested reader not already versed can see Sample Size Calculator for some quick experimentation, and Required sample sizes for hypothesis tests in the Wikipedia article Sample Size for a quick brush up on the basics.

Key to these facts is that the sample is unbiased and random. Otherwise, all bets are off. The introduction of of bias can be subtle. Say for example, you are thinking of producing a new widget that improves the flavor of coffee. You're going to take the world by storm! You want to get an idea of how many people would buy it, so you construct a survey, randomly picking a few hundred home phone numbers from the phone book of a random city with a fairly large population. Over a period of a few work days, your staff calls these random possible buyers, and queries their interest. Much to your dismay, very, very few have an interest. Was the survey proper? Can you infer that you might want to rethink your widget idea from the results?

Did you catch the two (and there of course are more) possibly fatal biases introduced by the survey? Firstly, calls were made during 'work days'. Perhaps the most interested prospective buyers work during the day, so you completely missed them in the results. Secondly, just what was the 'random' city? If it turned out to be say Salt Lake City, with a core and suburb population of well over 50% Mormon, a faith that many followers believe precludes the drinking of coffee (actually, from The Book of Mormon: "And again, hot drinks are not for the body or belly."), your results are going to be wildly biased.

So much for an unbiased and random survey.

The posters in a typical game enthusiast forum produce a vastly more biased view of the population than even this woefully inadequate example survey. This is because they are self-selecting. They are there by choice, not because they were randomly selected from the population. Importantly, a high percentage of these self-selected forum members only make posts when looking for solutions to problems, game caused or otherwise, or to state some other dissatisfaction with the product. Self-selection introduces one type of Selection Bias that makes any kind of valid statistical inference nearly impossible, even after adjusting or weighting the data.

One of the more interesting studies related to this kind of problem is covered in the landmark study Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples by David S. Yeager and Jon A. Krosnick of Stanford University. In it, the authors state:

"If the factors that determine a population member’s presence or absence in the sample are all uncorrelated with variables of interest in a study, then the observed distributions of those variables should be identical to the distributions in the population. And if the factors that determine a population member’s presence or absence in the sample are all uncorrelated with the magnitudes of associations between pairs of variables measured in the study, then the observed associations should also be identical to those in the population. However, if these conditions do not hold, then survey results may not be comparable to those that would be obtained from probability samples...More generally, the present investigation suggests that the foundations of statistical sampling theory are sustained by actual data in practice. Probability samples, even ones without especially high response rates, yield quite accurate results. In contrast, non-probability samples are not as accurate, and are sometimes strikingly inaccurate."

Quite the mouthful! Nonetheless, it cuts to the quick of the problem of trying to project the experiences of forum posters with game problems onto the general population of players of the game.

You can't, really. And doing so in a forum rant probably does nothing more than make one look like an Internet Jackass.