For Post Hoc's Sake: Determining Sample Size for Tukey Multiple Comparisons in 4-Group ANOVA

The determination of an appropriate sample size is a difficult, yet critically important, element in the research design process.Sample sizes in ANOVA are most often based on an overall standardized difference in the means, but these recommended sample sizes provide statistical power only for the omnibus F test.Adequate sample size for the omnibus test does not necessarily provide sufficient statistical power for the post hoc multiple comparisons typically performed following a statistically significant omnibus F test.The purpose of this paper is to use Monte Carlo techniques to determine sample sizes that will provide adequate statistical power for Tukey post hoc multiple comparison procedure (MCP) in 4-group ANOVA.Sample size tables are included.n the planning stage of a research study, investigators are often uncertain about the minimum number of subjects needed to adequately test a hypothesis of interest (Olejnik, 1984).Since Cohen proposed sample size determination in the planning, execution and interpretation of statistical analysis, researchers have increasingly realized the importance of sample size (Brewer & Sindelar, 1988).The number of cases that should be involved in a research study, particularly in hypothesis testing, has become a critical concern (Olejnik, 1984).Most researchers generally know that the sample size determination is functionally related to the significance level, statistical power, and the effect size.The required sample size is inversely related to significance level, given that power and effect size are held constant.A priori power, the probability of rejecting a null hypothesis that is indeed false, will inform researchers how many subjects will be needed for adequate power (Light, Singer, & Willett, 1990).Effect size is the degree to which the null hypothesis is false (Olejnik, 1984).Different effect size values will require different sample sizes.In addition, Brooks and Johanson (2011) pointed out that the sample size needed for an adequate test of a hypothesis is affected by the particular statistical analysis strategy that researchers choose.For example, when ANOVA will be used, adequate sample size for the omnibus test does not necessarily provide adequate statistical power for the post hoc multiple comparisons typically performed in ANOVA.This paper aims to demonstrate how to choose the sample size that will provide adequate power in the Tukey post hoc multiple comparison procedure (MCP).Pairwise multiple comparisons among four groups will be studied and results will be organized into recommended sample size tables for Tukey test.