Scale development with small samples: a new application of longitudinal item response theory

Carrie R. Houts, Robert Morlock, Steven I. Blum, Michael Edwards, R. J. Wirth

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

Purpose: Measurement development in hard-to-reach populations can pose methodological challenges. Item response theory (IRT) is a useful statistical tool, but often requires large samples. We describe the use of longitudinal IRT models as a pragmatic approach to instrument development when large samples are not feasible. Methods: The statistical foundations and practical benefits of longitudinal IRT models are briefly described. Results from a simulation study are reported to demonstrate the model’s ability to recover the generating measurement structure and parameters using a range of sample sizes, number of items, and number of time points. An example using early-phase clinical trial data in a rare condition demonstrates these methods in practice. Results: Simulation study results demonstrate that the longitudinal IRT model’s ability to recover the generating parameters rests largely on the interaction between sample size and the number of time points. Overall, the model performs well even in small samples provided a sufficient number of time points are available. The clinical trial data example demonstrates that by using conditional, longitudinal IRT models researchers can obtain stable estimates of psychometric characteristics from samples typically considered too small for rigorous psychometric modeling. Conclusion: Capitalizing on repeated measurements, it is possible to estimate psychometric characteristics for an assessment even when sample size is small. This allows researchers to optimize study designs and have increased confidence in subsequent comparisons using scores obtained from such models. While there are limitations and caveats to consider when using these models, longitudinal IRT modeling may be especially beneficial when developing measures for rare conditions and diseases in difficult-to-reach populations.

Original languageEnglish (US)
Pages (from-to)1721-1734
Number of pages14
JournalQuality of Life Research
Volume27
Issue number7
DOIs
StatePublished - Jul 1 2018

Fingerprint

Psychometrics
Sample Size
Aptitude
Research Personnel
Clinical Trials
Rare Diseases
Population

Keywords

  • Item response theory
  • Longitudinal data
  • Rare diseases
  • Scale development
  • Small sample statistics

ASJC Scopus subject areas

  • Public Health, Environmental and Occupational Health

Cite this

Scale development with small samples : a new application of longitudinal item response theory. / Houts, Carrie R.; Morlock, Robert; Blum, Steven I.; Edwards, Michael; Wirth, R. J.

In: Quality of Life Research, Vol. 27, No. 7, 01.07.2018, p. 1721-1734.

Research output: Contribution to journalArticle

Houts, Carrie R. ; Morlock, Robert ; Blum, Steven I. ; Edwards, Michael ; Wirth, R. J. / Scale development with small samples : a new application of longitudinal item response theory. In: Quality of Life Research. 2018 ; Vol. 27, No. 7. pp. 1721-1734.
@article{eb95d30ce0c84b7f8a8024622be2d779,
title = "Scale development with small samples: a new application of longitudinal item response theory",
abstract = "Purpose: Measurement development in hard-to-reach populations can pose methodological challenges. Item response theory (IRT) is a useful statistical tool, but often requires large samples. We describe the use of longitudinal IRT models as a pragmatic approach to instrument development when large samples are not feasible. Methods: The statistical foundations and practical benefits of longitudinal IRT models are briefly described. Results from a simulation study are reported to demonstrate the model’s ability to recover the generating measurement structure and parameters using a range of sample sizes, number of items, and number of time points. An example using early-phase clinical trial data in a rare condition demonstrates these methods in practice. Results: Simulation study results demonstrate that the longitudinal IRT model’s ability to recover the generating parameters rests largely on the interaction between sample size and the number of time points. Overall, the model performs well even in small samples provided a sufficient number of time points are available. The clinical trial data example demonstrates that by using conditional, longitudinal IRT models researchers can obtain stable estimates of psychometric characteristics from samples typically considered too small for rigorous psychometric modeling. Conclusion: Capitalizing on repeated measurements, it is possible to estimate psychometric characteristics for an assessment even when sample size is small. This allows researchers to optimize study designs and have increased confidence in subsequent comparisons using scores obtained from such models. While there are limitations and caveats to consider when using these models, longitudinal IRT modeling may be especially beneficial when developing measures for rare conditions and diseases in difficult-to-reach populations.",
keywords = "Item response theory, Longitudinal data, Rare diseases, Scale development, Small sample statistics",
author = "Houts, {Carrie R.} and Robert Morlock and Blum, {Steven I.} and Michael Edwards and Wirth, {R. J.}",
year = "2018",
month = "7",
day = "1",
doi = "10.1007/s11136-018-1801-z",
language = "English (US)",
volume = "27",
pages = "1721--1734",
journal = "Quality of Life Research",
issn = "0962-9343",
publisher = "Springer Netherlands",
number = "7",

}

TY - JOUR

T1 - Scale development with small samples

T2 - a new application of longitudinal item response theory

AU - Houts, Carrie R.

AU - Morlock, Robert

AU - Blum, Steven I.

AU - Edwards, Michael

AU - Wirth, R. J.

PY - 2018/7/1

Y1 - 2018/7/1

N2 - Purpose: Measurement development in hard-to-reach populations can pose methodological challenges. Item response theory (IRT) is a useful statistical tool, but often requires large samples. We describe the use of longitudinal IRT models as a pragmatic approach to instrument development when large samples are not feasible. Methods: The statistical foundations and practical benefits of longitudinal IRT models are briefly described. Results from a simulation study are reported to demonstrate the model’s ability to recover the generating measurement structure and parameters using a range of sample sizes, number of items, and number of time points. An example using early-phase clinical trial data in a rare condition demonstrates these methods in practice. Results: Simulation study results demonstrate that the longitudinal IRT model’s ability to recover the generating parameters rests largely on the interaction between sample size and the number of time points. Overall, the model performs well even in small samples provided a sufficient number of time points are available. The clinical trial data example demonstrates that by using conditional, longitudinal IRT models researchers can obtain stable estimates of psychometric characteristics from samples typically considered too small for rigorous psychometric modeling. Conclusion: Capitalizing on repeated measurements, it is possible to estimate psychometric characteristics for an assessment even when sample size is small. This allows researchers to optimize study designs and have increased confidence in subsequent comparisons using scores obtained from such models. While there are limitations and caveats to consider when using these models, longitudinal IRT modeling may be especially beneficial when developing measures for rare conditions and diseases in difficult-to-reach populations.

AB - Purpose: Measurement development in hard-to-reach populations can pose methodological challenges. Item response theory (IRT) is a useful statistical tool, but often requires large samples. We describe the use of longitudinal IRT models as a pragmatic approach to instrument development when large samples are not feasible. Methods: The statistical foundations and practical benefits of longitudinal IRT models are briefly described. Results from a simulation study are reported to demonstrate the model’s ability to recover the generating measurement structure and parameters using a range of sample sizes, number of items, and number of time points. An example using early-phase clinical trial data in a rare condition demonstrates these methods in practice. Results: Simulation study results demonstrate that the longitudinal IRT model’s ability to recover the generating parameters rests largely on the interaction between sample size and the number of time points. Overall, the model performs well even in small samples provided a sufficient number of time points are available. The clinical trial data example demonstrates that by using conditional, longitudinal IRT models researchers can obtain stable estimates of psychometric characteristics from samples typically considered too small for rigorous psychometric modeling. Conclusion: Capitalizing on repeated measurements, it is possible to estimate psychometric characteristics for an assessment even when sample size is small. This allows researchers to optimize study designs and have increased confidence in subsequent comparisons using scores obtained from such models. While there are limitations and caveats to consider when using these models, longitudinal IRT modeling may be especially beneficial when developing measures for rare conditions and diseases in difficult-to-reach populations.

KW - Item response theory

KW - Longitudinal data

KW - Rare diseases

KW - Scale development

KW - Small sample statistics

UR - http://www.scopus.com/inward/record.url?scp=85041527075&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041527075&partnerID=8YFLogxK

U2 - 10.1007/s11136-018-1801-z

DO - 10.1007/s11136-018-1801-z

M3 - Article

C2 - 29423756

AN - SCOPUS:85041527075

VL - 27

SP - 1721

EP - 1734

JO - Quality of Life Research

JF - Quality of Life Research

SN - 0962-9343

IS - 7

ER -