Z is for Z Test

Today we discuss the Wonderful World of Z Tests. I started hearing from co-workers last year, about testing means with Z-tests, and I wondered why one would use a Z-test when one has perfectly good t-tests. Z-tests can also be used to test proportions, such as people who prefer Coke vs. Pepsi. Why not use a Chi-square in this case?

The main reason to use Z-tests is when you have a population's mean or proportion of responses. Thus, you are learning about a particular sample by testing against a known universe of scores. Afterwards, you are completely done: you are not going to try to apply what you learned about this sample to other situations. When we use t-tests and chi-squares, we're doing inferential testing so that we can apply our test results to future samples. Think drug tests, where we want to know whether a drug is truly effective for a trial group so that it can be used on other patients.

Example A: At eAcmeWidget.com, they ask customers about their annual income when purchasing a widget. They want to know whether their customers are the same, richer, or poorer than the national average. To do this, they have to compare their customers' mean income against the national mean income, which they know from the last national census. To learn more about this type of Z-test, see this tutorial video about Z-test for the mean.

Usually, however, we don't know population means for the things we study in user experience. We might have time scores for users installing a program on their computer, but we don't have a national average to compare with.

Example B: Suppose an online multiplayer game called Funhouse lets the user play as either Bozo or JP Patches. A spin-off game, Funhouse 2: The Revenge, has the same two character choices but with a different story. The company wants to know whether JP Patches is as popular in Funhouse 2 as in the original Funhouse, based on which character users choose to play as.
Out of the 1,000 Funhouse players, 750 choose to play as JP Patches.
Out of the 500 Funhouse 2 players, 300 choose to play as JP Patches.
A Z-test of proportions shows that JP Patches is significantly less popular in Funhouse 2 than the original Funhouse. To calculate this kind of test, see the Z-test for two proportions calculator

In user experience, we would have more opportunities to run this type of Z-test on our various Yes/No and preference questions. Anything with two answer choices where you do not need to extrapolate to other samples would be fair game.