Evaluating long-term complex professional development: Using a variation of the cohort control design

This paper introduces a variation on the post-test only cohort control design and addresses questions concerning both the methodological credibility and the practical utility of employing this design variation in evaluations of large-scale complex professional development programmes in mathematics education. The original design and design variation, which adds a pre-test measure and gain scores for both pre-treatment and post-treatment conditions, are compared theoretically for their respective controls on threats to internal validity, and practically for their comparative ease of implementation in field settings. We conclude that the design variation adds important controls to selection bias that the original design cannot. From a utility perspective, however, there are tradeoffs, because this design variation requires more data collection observations, more complex data management and certain psychometric characteristics for the outcome measure.