What Would It Take to Change an Inference? Using Rubin's Causal Model to Interpret the Robustness of Causal Inferences

Kenneth A. Frank, Spiro Maroulis, Minh Q. Duong, Benjamin M. Kelcey

Research output: Contribution to journalArticle

42 Citations (Scopus)

Abstract

We contribute to debate about causal inferences in educational research in two ways. First, we quantify how much bias there must be in an estimate to invalidate an inference. Second, we utilize Rubin's causal model to interpret the bias necessary to invalidate an inference in terms of sample replacement. We apply our analysis to an inference of a positive effect of Open Court Curriculum on reading achievement from a randomized experiment, and an inference of a negative effect of kindergarten retention on reading achievement from an observational study. We consider details of our framework, and then discuss how our approach informs judgment of inference relative to study design. We conclude with implications for scientific discourse.

Original languageEnglish (US)
Pages (from-to)437-460
Number of pages24
JournalEducational Evaluation and Policy Analysis
Volume35
Issue number4
DOIs
StatePublished - Dec 2013

Fingerprint

trend
educational research
kindergarten
curriculum
discourse
experiment

Keywords

  • causal inference
  • observational studies
  • Rubin's causal model
  • sensitivity analysis

ASJC Scopus subject areas

  • Education

Cite this

What Would It Take to Change an Inference? Using Rubin's Causal Model to Interpret the Robustness of Causal Inferences. / Frank, Kenneth A.; Maroulis, Spiro; Duong, Minh Q.; Kelcey, Benjamin M.

In: Educational Evaluation and Policy Analysis, Vol. 35, No. 4, 12.2013, p. 437-460.

Research output: Contribution to journalArticle

@article{78e80dd3ba274f5aa370839739fa1d44,
title = "What Would It Take to Change an Inference? Using Rubin's Causal Model to Interpret the Robustness of Causal Inferences",
abstract = "We contribute to debate about causal inferences in educational research in two ways. First, we quantify how much bias there must be in an estimate to invalidate an inference. Second, we utilize Rubin's causal model to interpret the bias necessary to invalidate an inference in terms of sample replacement. We apply our analysis to an inference of a positive effect of Open Court Curriculum on reading achievement from a randomized experiment, and an inference of a negative effect of kindergarten retention on reading achievement from an observational study. We consider details of our framework, and then discuss how our approach informs judgment of inference relative to study design. We conclude with implications for scientific discourse.",
keywords = "causal inference, observational studies, Rubin's causal model, sensitivity analysis",
author = "Frank, {Kenneth A.} and Spiro Maroulis and Duong, {Minh Q.} and Kelcey, {Benjamin M.}",
year = "2013",
month = "12",
doi = "10.3102/0162373713493129",
language = "English (US)",
volume = "35",
pages = "437--460",
journal = "Educational Evaluation and Policy Analysis",
issn = "0162-3737",
publisher = "SAGE Publications Inc.",
number = "4",

}

TY - JOUR

T1 - What Would It Take to Change an Inference? Using Rubin's Causal Model to Interpret the Robustness of Causal Inferences

AU - Frank, Kenneth A.

AU - Maroulis, Spiro

AU - Duong, Minh Q.

AU - Kelcey, Benjamin M.

PY - 2013/12

Y1 - 2013/12

N2 - We contribute to debate about causal inferences in educational research in two ways. First, we quantify how much bias there must be in an estimate to invalidate an inference. Second, we utilize Rubin's causal model to interpret the bias necessary to invalidate an inference in terms of sample replacement. We apply our analysis to an inference of a positive effect of Open Court Curriculum on reading achievement from a randomized experiment, and an inference of a negative effect of kindergarten retention on reading achievement from an observational study. We consider details of our framework, and then discuss how our approach informs judgment of inference relative to study design. We conclude with implications for scientific discourse.

AB - We contribute to debate about causal inferences in educational research in two ways. First, we quantify how much bias there must be in an estimate to invalidate an inference. Second, we utilize Rubin's causal model to interpret the bias necessary to invalidate an inference in terms of sample replacement. We apply our analysis to an inference of a positive effect of Open Court Curriculum on reading achievement from a randomized experiment, and an inference of a negative effect of kindergarten retention on reading achievement from an observational study. We consider details of our framework, and then discuss how our approach informs judgment of inference relative to study design. We conclude with implications for scientific discourse.

KW - causal inference

KW - observational studies

KW - Rubin's causal model

KW - sensitivity analysis

UR - http://www.scopus.com/inward/record.url?scp=84886053476&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84886053476&partnerID=8YFLogxK

U2 - 10.3102/0162373713493129

DO - 10.3102/0162373713493129

M3 - Article

VL - 35

SP - 437

EP - 460

JO - Educational Evaluation and Policy Analysis

JF - Educational Evaluation and Policy Analysis

SN - 0162-3737

IS - 4

ER -