Abstract

Replication research holds an increasingly important place in modern psychological science. If such work is to improve the state of knowledge rather than add confusion, however, replication attempts must be held to high standards of rigor. As an example of how replication attempts can add confusion rather than clarity, we consider an article by Shanks and colleagues (2015). They conducted a meta-analysis of studies examining romantic motivation, using problematic criteria for the inclusion of effects and reached conclusions of a null effect that were unwarranted. A more rigorous and defensible approach, relying on a representative analysis of effects and p-curves, suggests a different, more positive conclusion with no evidence of p-hacking. Shanks et al. also conducted several experiments that suffered from numerous issues, such as relying on inappropriate subject samples (e.g., older adults likely to be less sensitive to mating manipulations than college students used in previous research), altered research methods, and demonstrably weak manipulations, among other problems. We discuss the broader implications of this case, to illustrate both the opportunities and the pitfalls inherent in attempts to replicate contextually sensitive research.

Original languageEnglish (US)
Pages (from-to)e1-e11
JournalJournal of Experimental Psychology: General
Volume148
Issue number4
DOIs
StatePublished - Apr 1 2019

Fingerprint

Research
Meta-Analysis
Students
Psychology

Keywords

  • Decision-making
  • Mating motivation
  • Meta-analysis
  • p-curve
  • Replication research

ASJC Scopus subject areas

  • Experimental and Cognitive Psychology
  • Psychology(all)
  • Developmental Neuroscience

Cite this

Moving beyond unwise replication practices : The case of romantic motivation. / Sundie, Jill M.; Beal, Daniel J.; Neuberg, Steven; Kenrick, Douglas.

In: Journal of Experimental Psychology: General, Vol. 148, No. 4, 01.04.2019, p. e1-e11.

Research output: Contribution to journalComment/debate

@article{4747e337d46848fb99060502bc375154,
title = "Moving beyond unwise replication practices: The case of romantic motivation",
abstract = "Replication research holds an increasingly important place in modern psychological science. If such work is to improve the state of knowledge rather than add confusion, however, replication attempts must be held to high standards of rigor. As an example of how replication attempts can add confusion rather than clarity, we consider an article by Shanks and colleagues (2015). They conducted a meta-analysis of studies examining romantic motivation, using problematic criteria for the inclusion of effects and reached conclusions of a null effect that were unwarranted. A more rigorous and defensible approach, relying on a representative analysis of effects and p-curves, suggests a different, more positive conclusion with no evidence of p-hacking. Shanks et al. also conducted several experiments that suffered from numerous issues, such as relying on inappropriate subject samples (e.g., older adults likely to be less sensitive to mating manipulations than college students used in previous research), altered research methods, and demonstrably weak manipulations, among other problems. We discuss the broader implications of this case, to illustrate both the opportunities and the pitfalls inherent in attempts to replicate contextually sensitive research.",
keywords = "Decision-making, Mating motivation, Meta-analysis, p-curve, Replication research",
author = "Sundie, {Jill M.} and Beal, {Daniel J.} and Steven Neuberg and Douglas Kenrick",
year = "2019",
month = "4",
day = "1",
doi = "10.1037/xge0000527",
language = "English (US)",
volume = "148",
pages = "e1--e11",
journal = "Journal of Experimental Psychology: General",
issn = "0096-3445",
publisher = "American Psychological Association Inc.",
number = "4",

}

TY - JOUR

T1 - Moving beyond unwise replication practices

T2 - The case of romantic motivation

AU - Sundie, Jill M.

AU - Beal, Daniel J.

AU - Neuberg, Steven

AU - Kenrick, Douglas

PY - 2019/4/1

Y1 - 2019/4/1

N2 - Replication research holds an increasingly important place in modern psychological science. If such work is to improve the state of knowledge rather than add confusion, however, replication attempts must be held to high standards of rigor. As an example of how replication attempts can add confusion rather than clarity, we consider an article by Shanks and colleagues (2015). They conducted a meta-analysis of studies examining romantic motivation, using problematic criteria for the inclusion of effects and reached conclusions of a null effect that were unwarranted. A more rigorous and defensible approach, relying on a representative analysis of effects and p-curves, suggests a different, more positive conclusion with no evidence of p-hacking. Shanks et al. also conducted several experiments that suffered from numerous issues, such as relying on inappropriate subject samples (e.g., older adults likely to be less sensitive to mating manipulations than college students used in previous research), altered research methods, and demonstrably weak manipulations, among other problems. We discuss the broader implications of this case, to illustrate both the opportunities and the pitfalls inherent in attempts to replicate contextually sensitive research.

AB - Replication research holds an increasingly important place in modern psychological science. If such work is to improve the state of knowledge rather than add confusion, however, replication attempts must be held to high standards of rigor. As an example of how replication attempts can add confusion rather than clarity, we consider an article by Shanks and colleagues (2015). They conducted a meta-analysis of studies examining romantic motivation, using problematic criteria for the inclusion of effects and reached conclusions of a null effect that were unwarranted. A more rigorous and defensible approach, relying on a representative analysis of effects and p-curves, suggests a different, more positive conclusion with no evidence of p-hacking. Shanks et al. also conducted several experiments that suffered from numerous issues, such as relying on inappropriate subject samples (e.g., older adults likely to be less sensitive to mating manipulations than college students used in previous research), altered research methods, and demonstrably weak manipulations, among other problems. We discuss the broader implications of this case, to illustrate both the opportunities and the pitfalls inherent in attempts to replicate contextually sensitive research.

KW - Decision-making

KW - Mating motivation

KW - Meta-analysis

KW - p-curve

KW - Replication research

UR - http://www.scopus.com/inward/record.url?scp=85061797917&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061797917&partnerID=8YFLogxK

U2 - 10.1037/xge0000527

DO - 10.1037/xge0000527

M3 - Comment/debate

VL - 148

SP - e1-e11

JO - Journal of Experimental Psychology: General

JF - Journal of Experimental Psychology: General

SN - 0096-3445

IS - 4

ER -