Experimental Designs That Use Fewer Subjects
I am trying to develop a drug abuse treatment intervention. There are six distinct components I am considering including in the intervention. I need to make the intervention as short as I can, so I don’t want to include any components that aren’t having much of an effect. I decided to conduct six experiments, each of which would examine the effect of a single intervention component. I need to be able to detect an effect size that is at least d =.3; any smaller than that and the component would not be pulling its own weight, so to speak. I have determined that I need a sample size of about N=200 for each experiment to maintain power at about .8. But then I did the math and figured out that with six experiments, I would need 6 X 200 = 1200 subjects! Yikes! Is there any way I can learn what I need to know, but using fewer subjects? —Signed, Experimental Design Gives Yips
Dear EDGY, Have you considered examining all six components in a single factorial experiment? If you did this, you would need only 200 subjects total to maintain power at .8 for testing the main effect of each component. This experiment would have 26=64 experimental conditions, with only a few subjects per condition. (In practice, you might want to use either 192 or 256 subjects so you could have equal numbers of subjects in each condition.) Unlike the approach of separate individual experiments, this factorial experiment would enable estimation of interactions.
Of course, an important question is whether it is feasible for you to implement 64 experimental conditions.You do not mention the context in which you are conducting this research. For example, if this is an internet-delivered intervention, 64 conditions may be feasible.
If it is not feasible for you to conduct an experiment with 64 conditions, but you would nevertheless like to take advantage of the economy associated with factorial experiments, you may want to consider conducting a fractional factorial experiment. Fractional factorial designs are used commonly in engineering. In general, these designs are most useful when there is no a prior reason to believe that the higher-order interactions are sizeable. There are a lot of different fractional factorial designs to choose from, each representing a different set of trade-offs between efficiency and assumptions. For example, with six independent variables there is a fractional factorial design available to you that would involve only 16 experimental conditions. This design would enable you to estimate all six main effects, and requires the assumption that all interactions three-way and higher are negligible. This design has the same sample size requirements as the complete factorial experiment, that is, N=200 to maintain power at .8. Other fractional factorial designs are available to you that require less stringent assumptions.
For a comparison of the economy of factorial experiments vs. individual experiments, and a brief tutorial on fractional factorial designs, see Collins, Dziak, and Li (2009).