Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power

Samantha F. Anderson, Scott E. Maxwell

Research output: Contribution to journalArticlepeer-review

37 Scopus citations

Abstract

Psychology is undergoing a replication crisis. The discussion surrounding this crisis has centered on mistrust of previous findings. Researchers planning replication studies often use the original study sample effect size as the basis for sample size planning. However, this strategy ignores uncertainty and publication bias in estimated effect sizes, resulting in overly optimistic calculations. A psychologist who intends to obtain power of.80 in the replication study, and performs calculations accordingly, may have an actual power lower than.80. We performed simulations to reveal the magnitude of the difference between actual and intended power based on common sample size planning strategies and assessed the performance of methods that aim to correct for effect size uncertainty and/or bias. Our results imply that even if original studies reflect actual phenomena and were conducted in the absence of questionable research practices, popular approaches to designing replication studies may result in a low success rate, especially if the original study is underpowered. Methods correcting for bias and/or uncertainty generally had higher actual power, but were not a panacea for an underpowered original study. Thus, it becomes imperative that 1) original studies are adequately powered and 2) replication studies are designed with methods that are more likely to yield the intended level of power.

Original languageEnglish (US)
Pages (from-to)305-324
Number of pages20
JournalMultivariate Behavioral Research
Volume52
Issue number3
DOIs
StatePublished - May 4 2017
Externally publishedYes

Keywords

  • Effect size
  • power
  • replication
  • sample size planning

ASJC Scopus subject areas

  • Statistics and Probability
  • Experimental and Cognitive Psychology
  • Arts and Humanities (miscellaneous)

Fingerprint

Dive into the research topics of 'Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power'. Together they form a unique fingerprint.

Cite this