More on Power, Sample Size, and Power Calculations
Of the variables that affect power, you can see that sample size gives you the most flexibility, as people are not in the habit of changing either the alpha level of .05, nor can you switch from a two-tailed to a one-tailed test (the use of a one-tailed test has to be stated in your hypotheses), nor can they change the effect size reported in the literature. Recommendations for calculating sample size include performing a power analysis before conducting the study.
The real challenge in determining how many participants you need is obtaining the effect size. Howell (2013, pp. 234-235) points to three routes: 1) existing research 2) the researcher’s decision about the magnitude of difference between m1 (the mean under H0) and m2 (the mean under H1) that would be considered important and 3) conventions accepted in the literature, such as those by Cohen (1988) (see Chapter 3). In other words, the third route (values from Cohen or others) is used when you have no other way to make an estimate of the parameters you need, but is not preferred (Howell, 2013). Howell (2013) mentions that there are a number of online software packages (many of them free) for calculating power and points to G*Power (Faul, Erdfelder, Lang, & Buchner, 2007) as one such option. Howell (2013) reminds us that “power = 1 – b” (p. 230), and b is the probability of making a Type II error. Thus, if power is set at .80, the probability is 20% that a Type II error will occur. To decrease that probability, in the same experiment, we would have to increase the number of participants.
As Howell describes, whether you are able to do that depends on the ease/difficulty of obtaining the particular participants you need. College students are easy to obtain, whereas therapy patients might be substantially harder to obtain. Keppel (1982) also provides comments about performing a power analysis; while acknowledging some of the difficulties researchers may encounter while trying to determine power, he underscores the merit of doing so: “An estimate of power, no matter how approximate, gives us some degree of control over type II error” (p. 73). While further discussion of power analysis will not be covered in this text, it is important to understand that such analyses can be performed, especially with the help of online calculators. Increasingly, such analyses are expected of researchers. When you are unable to reject the null hypothesis in your experiment, a natural question would be whether you had sufficient power to adequately evaluate that hypothesis. A power calculation tells you that, but it should be done a prior.