Using Generalizability Theory to Inform Optimal Design for a Nursing Performance Assessment

Janet O’Brien, Marilyn Thompson, Debra Hagler

Research output: Contribution to journalArticle

Abstract

The promotion of competency of nurses and other health-care professionals is a goal shared by many stakeholders. In nursing, observation-based assessments are often better suited than paper-and-pencil tests for assessing many clinical abilities. Unfortunately, few instruments for simulation-based assessment of competency have been published that have undergone stringent reliability and validity evaluation. Reliability analyses typically involve some measure of rater agreement, but other sources of measurement error that affect reliability should also be considered. The purpose of this study is three-fold. First, using extant data collected from 18 nurses evaluated on 3 Scenarios by 3 Raters, we utilize generalizability (G) theory to examine the psychometric characteristics of the Nursing Performance Profile, a simulation-based instrument for assessing nursing competency. Results corroborated findings of previous studies of simulation-based assessments showing that obtaining desired score reliability requires substantially greater numbers of scenarios and/or raters. Second, we provide an illustrative exemplar of how G theory can be used to understand the relative magnitudes of sources of error variance—such as scenarios, raters, and items—and their interactions. Finally, we offer general recommendations for the design and psychometric study of simulation-based assessments in health-care contexts.

Original languageEnglish (US)
Pages (from-to)297-327
Number of pages31
JournalEvaluation and the Health Professions
Volume42
Issue number3
DOIs
StatePublished - Sep 1 2019

Fingerprint

Nursing Assessment
Nursing
Psychometrics
Research Design
Nurses
Delivery of Health Care
Aptitude
Reproducibility of Results
Observation

Keywords

  • generalizability theory
  • nursing
  • observation-based assessment
  • reliability analysis
  • simulation

ASJC Scopus subject areas

  • Health Policy

Cite this

Using Generalizability Theory to Inform Optimal Design for a Nursing Performance Assessment. / O’Brien, Janet; Thompson, Marilyn; Hagler, Debra.

In: Evaluation and the Health Professions, Vol. 42, No. 3, 01.09.2019, p. 297-327.

Research output: Contribution to journalArticle

@article{018efb7e0ef0476fae7c1b0badbcd5a1,
title = "Using Generalizability Theory to Inform Optimal Design for a Nursing Performance Assessment",
abstract = "The promotion of competency of nurses and other health-care professionals is a goal shared by many stakeholders. In nursing, observation-based assessments are often better suited than paper-and-pencil tests for assessing many clinical abilities. Unfortunately, few instruments for simulation-based assessment of competency have been published that have undergone stringent reliability and validity evaluation. Reliability analyses typically involve some measure of rater agreement, but other sources of measurement error that affect reliability should also be considered. The purpose of this study is three-fold. First, using extant data collected from 18 nurses evaluated on 3 Scenarios by 3 Raters, we utilize generalizability (G) theory to examine the psychometric characteristics of the Nursing Performance Profile, a simulation-based instrument for assessing nursing competency. Results corroborated findings of previous studies of simulation-based assessments showing that obtaining desired score reliability requires substantially greater numbers of scenarios and/or raters. Second, we provide an illustrative exemplar of how G theory can be used to understand the relative magnitudes of sources of error variance—such as scenarios, raters, and items—and their interactions. Finally, we offer general recommendations for the design and psychometric study of simulation-based assessments in health-care contexts.",
keywords = "generalizability theory, nursing, observation-based assessment, reliability analysis, simulation",
author = "Janet O’Brien and Marilyn Thompson and Debra Hagler",
year = "2019",
month = "9",
day = "1",
doi = "10.1177/0163278717735565",
language = "English (US)",
volume = "42",
pages = "297--327",
journal = "Evaluation and the Health Professions",
issn = "0163-2787",
publisher = "SAGE Publications Inc.",
number = "3",

}

TY - JOUR

T1 - Using Generalizability Theory to Inform Optimal Design for a Nursing Performance Assessment

AU - O’Brien, Janet

AU - Thompson, Marilyn

AU - Hagler, Debra

PY - 2019/9/1

Y1 - 2019/9/1

N2 - The promotion of competency of nurses and other health-care professionals is a goal shared by many stakeholders. In nursing, observation-based assessments are often better suited than paper-and-pencil tests for assessing many clinical abilities. Unfortunately, few instruments for simulation-based assessment of competency have been published that have undergone stringent reliability and validity evaluation. Reliability analyses typically involve some measure of rater agreement, but other sources of measurement error that affect reliability should also be considered. The purpose of this study is three-fold. First, using extant data collected from 18 nurses evaluated on 3 Scenarios by 3 Raters, we utilize generalizability (G) theory to examine the psychometric characteristics of the Nursing Performance Profile, a simulation-based instrument for assessing nursing competency. Results corroborated findings of previous studies of simulation-based assessments showing that obtaining desired score reliability requires substantially greater numbers of scenarios and/or raters. Second, we provide an illustrative exemplar of how G theory can be used to understand the relative magnitudes of sources of error variance—such as scenarios, raters, and items—and their interactions. Finally, we offer general recommendations for the design and psychometric study of simulation-based assessments in health-care contexts.

AB - The promotion of competency of nurses and other health-care professionals is a goal shared by many stakeholders. In nursing, observation-based assessments are often better suited than paper-and-pencil tests for assessing many clinical abilities. Unfortunately, few instruments for simulation-based assessment of competency have been published that have undergone stringent reliability and validity evaluation. Reliability analyses typically involve some measure of rater agreement, but other sources of measurement error that affect reliability should also be considered. The purpose of this study is three-fold. First, using extant data collected from 18 nurses evaluated on 3 Scenarios by 3 Raters, we utilize generalizability (G) theory to examine the psychometric characteristics of the Nursing Performance Profile, a simulation-based instrument for assessing nursing competency. Results corroborated findings of previous studies of simulation-based assessments showing that obtaining desired score reliability requires substantially greater numbers of scenarios and/or raters. Second, we provide an illustrative exemplar of how G theory can be used to understand the relative magnitudes of sources of error variance—such as scenarios, raters, and items—and their interactions. Finally, we offer general recommendations for the design and psychometric study of simulation-based assessments in health-care contexts.

KW - generalizability theory

KW - nursing

KW - observation-based assessment

KW - reliability analysis

KW - simulation

UR - http://www.scopus.com/inward/record.url?scp=85069669611&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85069669611&partnerID=8YFLogxK

U2 - 10.1177/0163278717735565

DO - 10.1177/0163278717735565

M3 - Article

VL - 42

SP - 297

EP - 327

JO - Evaluation and the Health Professions

JF - Evaluation and the Health Professions

SN - 0163-2787

IS - 3

ER -