Making predictions

We can describe a data set's mean and standard deviation with perfect certainty. However, sometimes we want to make conclusions that stretch beyond the data set, such as when we use a sample to represent a whole population. How do we know we didn't draw a sample of people who like X or perform Y purely by chance? For these cases, we use inferential stats, which include ANOVA, regression, correlations, chi-square, MANOVA, and more. The general premise of inferential stats is to say with a level of confidence whether a phenomenon occurred because of a relationship between our variables or complete chance.

Let's say we asked 250 people on the street whether they preferred Search Engine A or B. We would expect the answers to split about 50/50 if the search engines did not differ from each other. Using an inferential statistic, we could test whether our responses of 150 people who prefer Search Engine A and 100 Search Engine B were due to chance or an actual preference for Search Engine A.

As part of this test we would state our tolerance for error or the level that we require for statistical significance. In social sciences, the alpha level, or bar for statistical significance, is often 0.05, that is, we are willing to accept a 5% chance that we incorrectly believe that people have a preference for Search Engine A. The tolerance for error will change depending on the test context. A clinical trial of a new drug will probably have lower tolerance for error than a taste test of chicken nuggets.

In this example, we run a one-way chi-squre, which tells us whether the preference for Search Engine A was statistically significant. If significant, the p value (probability value) would be less than what we set as the alpha level. The results of a chi-square and any other inferential stat would have the statistical test score, in this case a χ², degrees of freedom and/or sample size in parentheses, and the level of probability (p). If appropriate, the means and standard deviations of the comparison groups would be reported. Below, the sample report shows the χ² value as 10 based on 1 degree of freedom and a sample of 250 scores, with less than 1 percent probability that the results were due to chance

Our street survey showed a statistically significant preference by women for Search Engine A, χ² (1, N=250) = 10, p < 0.01. Future research can investigate the design characteristics that make it more preferable Search Engine A more preferable than B.