Using Generalizability Theory to Inform Optimal Design for a Nursing Performance Assessment

Janet O’Brien, Marilyn S. Thompson, Debra Hagler

Research output: Contribution to journalArticle

Abstract

The promotion of competency of nurses and other health-care professionals is a goal shared by many stakeholders. In nursing, observation-based assessments are often better suited than paper-and-pencil tests for assessing many clinical abilities. Unfortunately, few instruments for simulation-based assessment of competency have been published that have undergone stringent reliability and validity evaluation. Reliability analyses typically involve some measure of rater agreement, but other sources of measurement error that affect reliability should also be considered. The purpose of this study is three-fold. First, using extant data collected from 18 nurses evaluated on 3 Scenarios by 3 Raters, we utilize generalizability (G) theory to examine the psychometric characteristics of the Nursing Performance Profile, a simulation-based instrument for assessing nursing competency. Results corroborated findings of previous studies of simulation-based assessments showing that obtaining desired score reliability requires substantially greater numbers of scenarios and/or raters. Second, we provide an illustrative exemplar of how G theory can be used to understand the relative magnitudes of sources of error variance—such as scenarios, raters, and items—and their interactions. Finally, we offer general recommendations for the design and psychometric study of simulation-based assessments in health-care contexts.

Original languageEnglish (US)
Pages (from-to)297-327
Number of pages31
JournalEvaluation and the Health Professions
Volume42
Issue number3
DOIs
StatePublished - Sep 1 2019

Keywords

  • generalizability theory
  • nursing
  • observation-based assessment
  • reliability analysis
  • simulation

ASJC Scopus subject areas

  • Health Policy

Fingerprint Dive into the research topics of 'Using Generalizability Theory to Inform Optimal Design for a Nursing Performance Assessment'. Together they form a unique fingerprint.

  • Cite this