Representative and Quasi-Representative Designs for Research on Teaching

A decade has passed since the publication of Campbell and Stanley's (1963) monumental treatise on experimental design in educational research. That document synthesized much prior thinking as well as original work and established clear principles for both designing and evaluating educational experiments. It provided a systematic study of quasi-experiments, i.e., useful compromises on the characteristics of true experimental designs. It also did much to develop the experimental attitude so apparent today among experienced and novice investigators alike. In analyzing threats to the validity of experiments, Campbell and Stanley distinguished between internal validity (i.e., interpretability) and external validity (i.e., generalizability), but they concentrated attention on internal validity. An important extension of their work was contributed by Bracht and Glass (1968), who elaborated considerably the concept of external validity. They dealt with two classes of threats to external validity, calling these population validity and ecological validity. Under the first heading, they examined the issues involved in generalizing from samples, through experimentally accessible populations of students, to ultimate target populations and reviewed the possibility that person characteristics might interact with experimental variables to limit generalizations about treatment effects. Under ecological validity, they discussed the problems of describing independent variables, multiple-treatment interference, Hawthorne, novelty, and experimenter effects, pretest and posttest sensitization, interaction of history and treatment, and the measurement of dependent variables. They closed their paper suggesting that extension of the list of threats and revision of experimental designs to control for such threats would be useful next steps. This paper explores some of these next steps. Initially, it seeks merely to reconstruct and extend some old ideas for inclusion in the field that might be called the metatheory or comparative methodology of educational research. In so doing, however, it suggests that some rather unconventional additions and revisions in the design and analysis of educational experiments may be needed. It becomes apparent that the biggest threat to external validity may come when the experiment does not fit the nature of the behavior being studied and,