A Monte Carlo comparison of measures of relative and absolute monitoring accuracy

John L. Nietfeld, Craig K. Enders, Gregory Schraw

Research output: Contribution to journalArticle

31 Citations (Scopus)

Abstract

Researchers studying monitoring accuracy currently use two different indexes to estimate accuracy: relative accuracy and absolute accuracy. The authors compared the distributional properties of two measures of monitoring accuracy using Monte Carlo procedures that fit within these categories. They manipulated the accuracy of judgments (i.e., chance level or 60% and above) and the number of items per test (i.e., 20, 50, or 1,000) using 10,000 computer-generated cases. Gamma, an estimate of relative accuracy, yielded a skewed, leptokurtic distribution under the 50-item, 60% accuracy conditions. The Hamann coefficient, an estimate of absolute accuracy, yielded a normal distribution under the same conditions. Both statistics yielded normal distributions under the 1,000item, 60% conditions, although parameter estimates differed widely. The two statistics were similar, and normally distributed, under the 50and 1,000-item, chance conditions. Recommendations are made regarding the use of each measure in applied monitoring accuracy research.

Original languageEnglish (US)
Pages (from-to)258-271
Number of pages14
JournalEducational and Psychological Measurement
Volume66
Issue number2
DOIs
StatePublished - Apr 2006
Externally publishedYes

Fingerprint

Normal Distribution
Normal distribution
Monitoring
monitoring
statistics
Statistics
Research Personnel
Research
Estimate
Gaussian distribution
Leptokurtic
Recommendations

Keywords

  • Calibration
  • Metacognition
  • Monitoring accuracy
  • Monte Carlo

ASJC Scopus subject areas

  • Mathematics (miscellaneous)
  • Psychology(all)
  • Developmental and Educational Psychology
  • Psychology (miscellaneous)

Cite this

A Monte Carlo comparison of measures of relative and absolute monitoring accuracy. / Nietfeld, John L.; Enders, Craig K.; Schraw, Gregory.

In: Educational and Psychological Measurement, Vol. 66, No. 2, 04.2006, p. 258-271.

Research output: Contribution to journalArticle

Nietfeld, John L. ; Enders, Craig K. ; Schraw, Gregory. / A Monte Carlo comparison of measures of relative and absolute monitoring accuracy. In: Educational and Psychological Measurement. 2006 ; Vol. 66, No. 2. pp. 258-271.
@article{0fae968cbe1a4a3fb97185b696ef5f2d,
title = "A Monte Carlo comparison of measures of relative and absolute monitoring accuracy",
abstract = "Researchers studying monitoring accuracy currently use two different indexes to estimate accuracy: relative accuracy and absolute accuracy. The authors compared the distributional properties of two measures of monitoring accuracy using Monte Carlo procedures that fit within these categories. They manipulated the accuracy of judgments (i.e., chance level or 60{\%} and above) and the number of items per test (i.e., 20, 50, or 1,000) using 10,000 computer-generated cases. Gamma, an estimate of relative accuracy, yielded a skewed, leptokurtic distribution under the 50-item, 60{\%} accuracy conditions. The Hamann coefficient, an estimate of absolute accuracy, yielded a normal distribution under the same conditions. Both statistics yielded normal distributions under the 1,000item, 60{\%} conditions, although parameter estimates differed widely. The two statistics were similar, and normally distributed, under the 50and 1,000-item, chance conditions. Recommendations are made regarding the use of each measure in applied monitoring accuracy research.",
keywords = "Calibration, Metacognition, Monitoring accuracy, Monte Carlo",
author = "Nietfeld, {John L.} and Enders, {Craig K.} and Gregory Schraw",
year = "2006",
month = "4",
doi = "10.1177/0013164404273945",
language = "English (US)",
volume = "66",
pages = "258--271",
journal = "Educational and Psychological Measurement",
issn = "0013-1644",
publisher = "SAGE Publications Inc.",
number = "2",

}

TY - JOUR

T1 - A Monte Carlo comparison of measures of relative and absolute monitoring accuracy

AU - Nietfeld, John L.

AU - Enders, Craig K.

AU - Schraw, Gregory

PY - 2006/4

Y1 - 2006/4

N2 - Researchers studying monitoring accuracy currently use two different indexes to estimate accuracy: relative accuracy and absolute accuracy. The authors compared the distributional properties of two measures of monitoring accuracy using Monte Carlo procedures that fit within these categories. They manipulated the accuracy of judgments (i.e., chance level or 60% and above) and the number of items per test (i.e., 20, 50, or 1,000) using 10,000 computer-generated cases. Gamma, an estimate of relative accuracy, yielded a skewed, leptokurtic distribution under the 50-item, 60% accuracy conditions. The Hamann coefficient, an estimate of absolute accuracy, yielded a normal distribution under the same conditions. Both statistics yielded normal distributions under the 1,000item, 60% conditions, although parameter estimates differed widely. The two statistics were similar, and normally distributed, under the 50and 1,000-item, chance conditions. Recommendations are made regarding the use of each measure in applied monitoring accuracy research.

AB - Researchers studying monitoring accuracy currently use two different indexes to estimate accuracy: relative accuracy and absolute accuracy. The authors compared the distributional properties of two measures of monitoring accuracy using Monte Carlo procedures that fit within these categories. They manipulated the accuracy of judgments (i.e., chance level or 60% and above) and the number of items per test (i.e., 20, 50, or 1,000) using 10,000 computer-generated cases. Gamma, an estimate of relative accuracy, yielded a skewed, leptokurtic distribution under the 50-item, 60% accuracy conditions. The Hamann coefficient, an estimate of absolute accuracy, yielded a normal distribution under the same conditions. Both statistics yielded normal distributions under the 1,000item, 60% conditions, although parameter estimates differed widely. The two statistics were similar, and normally distributed, under the 50and 1,000-item, chance conditions. Recommendations are made regarding the use of each measure in applied monitoring accuracy research.

KW - Calibration

KW - Metacognition

KW - Monitoring accuracy

KW - Monte Carlo

UR - http://www.scopus.com/inward/record.url?scp=33644746350&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33644746350&partnerID=8YFLogxK

U2 - 10.1177/0013164404273945

DO - 10.1177/0013164404273945

M3 - Article

AN - SCOPUS:33644746350

VL - 66

SP - 258

EP - 271

JO - Educational and Psychological Measurement

JF - Educational and Psychological Measurement

SN - 0013-1644

IS - 2

ER -