Decision Making Using Data From a Factorial Experiment: Practice Data Sets
About the Practice Decision-Making Data Sets
Often in the optimization phase of MOST the investigator conducts a factorial experiment for the purpose of component screening. The results of the experiment form the basis for making decisions about what components and component levels to include in the optimized intervention.
In general, intervention scientists do not have much experience with this kind of decision making. In our work, we have found it extremely helpful to practice the decision making process using artificial data WELL BEFORE analyzing the empirical data. This provides a way of gaining a little experience in advance.
We have provided for your use three artificial data sets generated to resemble data from actual implementations of MOST. Here is how we recommend using each data set:
- Conduct the ANOVA on the data.
- Prepare plots of any interactions that are likely to be important in decision making.
- Hold a meeting of all decision makers, and go through the exercise of selecting components and component levels based on the ANOVA results and plots.
NOTE: Be sure to allow enough time – the decisions may not be straightforward. At a minimum, allow two hours.
The data sets and accompanying materials are supplements to a journal article that describes one approach to decision making. You can use the approach described there, or any rational approach. If you plan to use an optimization criterion that involves cost, you can make up data on the cost to implement each component level, and/or the time it takes to implement each component level, and incorporate this information into the decision making process.
We have provided ANOVA output and selected plots so you can check your results against ours. We also provide an “answer key,” which is simply a rank ordering of the experimental conditions by the true mean on the outcome variable. In real life this would be unknown, of course, but because these are artificial data we generated, in this case we know the true experimental condition means. We STRONGLY recommend that you do not look at the answer key until AFTER the decision making is complete.
Do not expect that your decision making process will lead you to the combination of components and levels that is ranked #1 in the answer key. The objective of MOST is NOT to identify the #1 ranked combination; rather, it is to enable the investigator to identify ONE OF THE TOP RANKED combinations, or, put another way, to weed out the poorly performing combinations. Moreover, the ranking in the answer key does not take cost into account.