Reliability and Model Fit

Leanne M. Stanley, Michael Edwards

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

The purpose of this article is to highlight the distinction between the reliability of test scores and the fit of psychometric measurement models, reminding readers why it is important to consider both when evaluating whether test scores are valid for a proposed interpretation and/or use. It is often the case that an investigator judges both the reliability of scores and the fit of a corresponding measurement model to be either acceptable or unacceptable for a given situation, but these are not the only possible outcomes. This article focuses on situations in which model fit is deemed acceptable, but reliability is not. Data were simulated based on the item characteristics of the PROMIS (Patient Reported Outcomes Measurement Information System) anxiety item bank and analyzed using methods from classical test theory, factor analysis, and item response theory. Analytic techniques from different psychometric traditions were used to illustrate that reliability and model fit are distinct, and that disagreement among indices of reliability and model fit may provide important information bearing on a particular validity argument, independent of the data analytic techniques chosen for a particular research application. We conclude by discussing the important information gleaned from the assessment of reliability and model fit.

Original languageEnglish (US)
Pages (from-to)976-985
Number of pages10
JournalEducational and Psychological Measurement
Volume76
Issue number6
DOIs
StatePublished - Dec 1 2016
Externally publishedYes

Fingerprint

Psychometrics
Information Systems
Statistical Factor Analysis
Anxiety
Score Test
Research Personnel
psychometrics
Bearings (structural)
Research
Model
Test Theory
test theory
Factor analysis
Factor Analysis
Measurement System
factor analysis
information system
bank
Information systems
Valid

Keywords

  • factor analysis
  • item response theory
  • model fit
  • reliability

ASJC Scopus subject areas

  • Education
  • Developmental and Educational Psychology
  • Applied Psychology
  • Applied Mathematics

Cite this

Reliability and Model Fit. / Stanley, Leanne M.; Edwards, Michael.

In: Educational and Psychological Measurement, Vol. 76, No. 6, 01.12.2016, p. 976-985.

Research output: Contribution to journalArticle

Stanley, Leanne M. ; Edwards, Michael. / Reliability and Model Fit. In: Educational and Psychological Measurement. 2016 ; Vol. 76, No. 6. pp. 976-985.
@article{93dac6f3f34d4f849217352d3ca2e3c3,
title = "Reliability and Model Fit",
abstract = "The purpose of this article is to highlight the distinction between the reliability of test scores and the fit of psychometric measurement models, reminding readers why it is important to consider both when evaluating whether test scores are valid for a proposed interpretation and/or use. It is often the case that an investigator judges both the reliability of scores and the fit of a corresponding measurement model to be either acceptable or unacceptable for a given situation, but these are not the only possible outcomes. This article focuses on situations in which model fit is deemed acceptable, but reliability is not. Data were simulated based on the item characteristics of the PROMIS (Patient Reported Outcomes Measurement Information System) anxiety item bank and analyzed using methods from classical test theory, factor analysis, and item response theory. Analytic techniques from different psychometric traditions were used to illustrate that reliability and model fit are distinct, and that disagreement among indices of reliability and model fit may provide important information bearing on a particular validity argument, independent of the data analytic techniques chosen for a particular research application. We conclude by discussing the important information gleaned from the assessment of reliability and model fit.",
keywords = "factor analysis, item response theory, model fit, reliability",
author = "Stanley, {Leanne M.} and Michael Edwards",
year = "2016",
month = "12",
day = "1",
doi = "10.1177/0013164416638900",
language = "English (US)",
volume = "76",
pages = "976--985",
journal = "Educational and Psychological Measurement",
issn = "0013-1644",
publisher = "SAGE Publications Inc.",
number = "6",

}

TY - JOUR

T1 - Reliability and Model Fit

AU - Stanley, Leanne M.

AU - Edwards, Michael

PY - 2016/12/1

Y1 - 2016/12/1

N2 - The purpose of this article is to highlight the distinction between the reliability of test scores and the fit of psychometric measurement models, reminding readers why it is important to consider both when evaluating whether test scores are valid for a proposed interpretation and/or use. It is often the case that an investigator judges both the reliability of scores and the fit of a corresponding measurement model to be either acceptable or unacceptable for a given situation, but these are not the only possible outcomes. This article focuses on situations in which model fit is deemed acceptable, but reliability is not. Data were simulated based on the item characteristics of the PROMIS (Patient Reported Outcomes Measurement Information System) anxiety item bank and analyzed using methods from classical test theory, factor analysis, and item response theory. Analytic techniques from different psychometric traditions were used to illustrate that reliability and model fit are distinct, and that disagreement among indices of reliability and model fit may provide important information bearing on a particular validity argument, independent of the data analytic techniques chosen for a particular research application. We conclude by discussing the important information gleaned from the assessment of reliability and model fit.

AB - The purpose of this article is to highlight the distinction between the reliability of test scores and the fit of psychometric measurement models, reminding readers why it is important to consider both when evaluating whether test scores are valid for a proposed interpretation and/or use. It is often the case that an investigator judges both the reliability of scores and the fit of a corresponding measurement model to be either acceptable or unacceptable for a given situation, but these are not the only possible outcomes. This article focuses on situations in which model fit is deemed acceptable, but reliability is not. Data were simulated based on the item characteristics of the PROMIS (Patient Reported Outcomes Measurement Information System) anxiety item bank and analyzed using methods from classical test theory, factor analysis, and item response theory. Analytic techniques from different psychometric traditions were used to illustrate that reliability and model fit are distinct, and that disagreement among indices of reliability and model fit may provide important information bearing on a particular validity argument, independent of the data analytic techniques chosen for a particular research application. We conclude by discussing the important information gleaned from the assessment of reliability and model fit.

KW - factor analysis

KW - item response theory

KW - model fit

KW - reliability

UR - http://www.scopus.com/inward/record.url?scp=84992560990&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84992560990&partnerID=8YFLogxK

U2 - 10.1177/0013164416638900

DO - 10.1177/0013164416638900

M3 - Article

AN - SCOPUS:84992560990

VL - 76

SP - 976

EP - 985

JO - Educational and Psychological Measurement

JF - Educational and Psychological Measurement

SN - 0013-1644

IS - 6

ER -