Interpreting Probabilistic Classifications From Diagnostic Psychometric Models

Laine Bradshaw, Roy Levy

Research output: Contribution to journalArticle

Abstract

Although much research has been conducted on the psychometric properties of cognitive diagnostic models, they are only recently being used in operational settings to provide results to examinees and other stakeholders. Using this newer class of models in practice comes with a fresh challenge for diagnostic assessment developers: effectively reporting results and supporting end users to accurately interpret results. Achieving the goal of communicating results in a way that leads users of the assessment to make accurate interpretations requires a prerequisite step that cannot be taken for granted. The assessment developers must first accurately interpret results from a psychometric, or measurement, standpoint. Through this article, we seek to begin a discussion about reasonable interpretations of the results that classification-based models provide about examinees. Interpretations from published research and ongoing practice show different—and sometimes conflicting—ways to interpret these results. This article seeks to formalize a comparison, critique, and discussion among the interpretations. Before beginning this discussion, we first present background on the results provided by classification-based models regarding the examinees. We then structure our discussion around key questions an assessment development team needs to answer themselves prior to constructing reports and interpretative guides for end users of the assessment.

Original languageEnglish (US)
Pages (from-to)79-88
Number of pages10
JournalEducational Measurement: Issues and Practice
Volume38
Issue number2
DOIs
StatePublished - Jun 1 2019

Fingerprint

psychometrics
diagnostic
interpretation
stakeholder
present

Keywords

  • cognitive diagnosis models
  • diagnostic classification models
  • diagnostic model reporting
  • probabilistic classification
  • score interpretation

ASJC Scopus subject areas

  • Education

Cite this

Interpreting Probabilistic Classifications From Diagnostic Psychometric Models. / Bradshaw, Laine; Levy, Roy.

In: Educational Measurement: Issues and Practice, Vol. 38, No. 2, 01.06.2019, p. 79-88.

Research output: Contribution to journalArticle

@article{27e400abb51947a0a6296f454644fa66,
title = "Interpreting Probabilistic Classifications From Diagnostic Psychometric Models",
abstract = "Although much research has been conducted on the psychometric properties of cognitive diagnostic models, they are only recently being used in operational settings to provide results to examinees and other stakeholders. Using this newer class of models in practice comes with a fresh challenge for diagnostic assessment developers: effectively reporting results and supporting end users to accurately interpret results. Achieving the goal of communicating results in a way that leads users of the assessment to make accurate interpretations requires a prerequisite step that cannot be taken for granted. The assessment developers must first accurately interpret results from a psychometric, or measurement, standpoint. Through this article, we seek to begin a discussion about reasonable interpretations of the results that classification-based models provide about examinees. Interpretations from published research and ongoing practice show different—and sometimes conflicting—ways to interpret these results. This article seeks to formalize a comparison, critique, and discussion among the interpretations. Before beginning this discussion, we first present background on the results provided by classification-based models regarding the examinees. We then structure our discussion around key questions an assessment development team needs to answer themselves prior to constructing reports and interpretative guides for end users of the assessment.",
keywords = "cognitive diagnosis models, diagnostic classification models, diagnostic model reporting, probabilistic classification, score interpretation",
author = "Laine Bradshaw and Roy Levy",
year = "2019",
month = "6",
day = "1",
doi = "10.1111/emip.12247",
language = "English (US)",
volume = "38",
pages = "79--88",
journal = "Educational Measurement: Issues and Practice",
issn = "0731-1745",
publisher = "Wiley-Blackwell",
number = "2",

}

TY - JOUR

T1 - Interpreting Probabilistic Classifications From Diagnostic Psychometric Models

AU - Bradshaw, Laine

AU - Levy, Roy

PY - 2019/6/1

Y1 - 2019/6/1

N2 - Although much research has been conducted on the psychometric properties of cognitive diagnostic models, they are only recently being used in operational settings to provide results to examinees and other stakeholders. Using this newer class of models in practice comes with a fresh challenge for diagnostic assessment developers: effectively reporting results and supporting end users to accurately interpret results. Achieving the goal of communicating results in a way that leads users of the assessment to make accurate interpretations requires a prerequisite step that cannot be taken for granted. The assessment developers must first accurately interpret results from a psychometric, or measurement, standpoint. Through this article, we seek to begin a discussion about reasonable interpretations of the results that classification-based models provide about examinees. Interpretations from published research and ongoing practice show different—and sometimes conflicting—ways to interpret these results. This article seeks to formalize a comparison, critique, and discussion among the interpretations. Before beginning this discussion, we first present background on the results provided by classification-based models regarding the examinees. We then structure our discussion around key questions an assessment development team needs to answer themselves prior to constructing reports and interpretative guides for end users of the assessment.

AB - Although much research has been conducted on the psychometric properties of cognitive diagnostic models, they are only recently being used in operational settings to provide results to examinees and other stakeholders. Using this newer class of models in practice comes with a fresh challenge for diagnostic assessment developers: effectively reporting results and supporting end users to accurately interpret results. Achieving the goal of communicating results in a way that leads users of the assessment to make accurate interpretations requires a prerequisite step that cannot be taken for granted. The assessment developers must first accurately interpret results from a psychometric, or measurement, standpoint. Through this article, we seek to begin a discussion about reasonable interpretations of the results that classification-based models provide about examinees. Interpretations from published research and ongoing practice show different—and sometimes conflicting—ways to interpret these results. This article seeks to formalize a comparison, critique, and discussion among the interpretations. Before beginning this discussion, we first present background on the results provided by classification-based models regarding the examinees. We then structure our discussion around key questions an assessment development team needs to answer themselves prior to constructing reports and interpretative guides for end users of the assessment.

KW - cognitive diagnosis models

KW - diagnostic classification models

KW - diagnostic model reporting

KW - probabilistic classification

KW - score interpretation

UR - http://www.scopus.com/inward/record.url?scp=85066932710&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066932710&partnerID=8YFLogxK

U2 - 10.1111/emip.12247

DO - 10.1111/emip.12247

M3 - Article

AN - SCOPUS:85066932710

VL - 38

SP - 79

EP - 88

JO - Educational Measurement: Issues and Practice

JF - Educational Measurement: Issues and Practice

SN - 0731-1745

IS - 2

ER -