You are here

Post Hoc Comparisons

30 November, 2015 - 16:43

When we reject the null hypothesis in a one-way ANOVA, we conclude that the group means are not all the same in the population. But this can indicate different things. With three groups, it can indicate that all three means are significantly different from each other. Or it can indicate that one of the means is significantly different from the other two, but the other two are not significantly different from each other. It could be, for example, that the mean calorie estimates of psychology majors, nutrition majors, and dieticians are all significantly different from each other. Or it could be that the mean for dieticians is significantly different from the means for psychology and nutrition majors, but the means for psychology and nutrition majors are not significantly different from each other. For this reason, statistically significant one-way ANOVA results are typically followed up with a series of poshocomparisonof selected pairs of group means to determine which are different from which others.

One approach to post hoc comparisons would be to conduct a series of independent-samples ttests comparing each group mean to each of the other group means. But there is a problem with this approach. In general, if we conduct a ttest when the null hypothesis is true, we have a 5% chance of mistakenly rejecting the null hypothesis (see Additional Considerations for more on such Type I errors). If we conduct several ttests when the null hypothesis is true, the chance of mistakenly rejecting at leastonenull hypothesis increases with each test we conduct. Thus researchers do not usually make post hoc comparisons using standard ttests because there is too great a chance that they will mistakenly reject at least one null hypothesis. Instead, they use one of several modified ttest procedures—among them the Bonferonni procedure, Fisher’s least significant difference (LSD) test, and Tukey’s honestly significant difference (HSD) test. The details of these approaches are beyond the scope of this book, but it is important to understand their purpose. It is to keep the risk of mistakenly rejecting a true null hypothesis to an acceptable level (close to 5%).