Improving the Measurement of School Climate Using Item Response Theory

Sarah Lindstrom Johnson, Ray E. Reichenberg, Kathan Shukla, Tracy E. Waasdorp, Catherine P. Bradshaw

Research output: Contribution to journalArticle

Abstract

The U.S. government has become increasingly focused on school climate, as recently evidenced by its inclusion as an accountability indicator in the Every Student Succeeds Act. Yet, there remains considerable variability in both conceptualizing and measuring school climate. To better inform the research and practice related to school climate and its measurement, we leveraged item response theory (IRT), a commonly used psychometric approach for the design of achievement assessments, to create a parsimonious measure of school climate that operates across varying individual characteristics. Students (n = 69,513) in 111 secondary schools completed a school climate assessment focused on three domains of climate (i.e., safety, engagement, and environment), as defined by the U.S. Department of Education. Item and test characteristics were estimated using the mirt package in R using unidimensional IRT. Analyses revealed measurement difficulties that resulted in a greater ability to assess less favorable perspectives on school climate. Differential item functioning analyses indicated measurement differences based on student academic success. These findings support the development of a broad measure of school climate but also highlight the importance of work to ensure precision in measuring school climate, particularly when considering use as an accountability measure.

Original languageEnglish (US)
JournalEducational Measurement: Issues and Practice
DOIs
StateAccepted/In press - Jan 1 2019
Externally publishedYes

Fingerprint

school climate
responsibility
student
academic success
psychometrics
secondary school
inclusion
climate
ability

Keywords

  • item response theory
  • measurement invariance
  • school accountability
  • school climate

ASJC Scopus subject areas

  • Education

Cite this

Improving the Measurement of School Climate Using Item Response Theory. / Lindstrom Johnson, Sarah; Reichenberg, Ray E.; Shukla, Kathan; Waasdorp, Tracy E.; Bradshaw, Catherine P.

In: Educational Measurement: Issues and Practice, 01.01.2019.

Research output: Contribution to journalArticle

Lindstrom Johnson, Sarah ; Reichenberg, Ray E. ; Shukla, Kathan ; Waasdorp, Tracy E. ; Bradshaw, Catherine P. / Improving the Measurement of School Climate Using Item Response Theory. In: Educational Measurement: Issues and Practice. 2019.
@article{e7ca0f1d4e3249e59a6f195cdbf106d3,
title = "Improving the Measurement of School Climate Using Item Response Theory",
abstract = "The U.S. government has become increasingly focused on school climate, as recently evidenced by its inclusion as an accountability indicator in the Every Student Succeeds Act. Yet, there remains considerable variability in both conceptualizing and measuring school climate. To better inform the research and practice related to school climate and its measurement, we leveraged item response theory (IRT), a commonly used psychometric approach for the design of achievement assessments, to create a parsimonious measure of school climate that operates across varying individual characteristics. Students (n = 69,513) in 111 secondary schools completed a school climate assessment focused on three domains of climate (i.e., safety, engagement, and environment), as defined by the U.S. Department of Education. Item and test characteristics were estimated using the mirt package in R using unidimensional IRT. Analyses revealed measurement difficulties that resulted in a greater ability to assess less favorable perspectives on school climate. Differential item functioning analyses indicated measurement differences based on student academic success. These findings support the development of a broad measure of school climate but also highlight the importance of work to ensure precision in measuring school climate, particularly when considering use as an accountability measure.",
keywords = "item response theory, measurement invariance, school accountability, school climate",
author = "{Lindstrom Johnson}, Sarah and Reichenberg, {Ray E.} and Kathan Shukla and Waasdorp, {Tracy E.} and Bradshaw, {Catherine P.}",
year = "2019",
month = "1",
day = "1",
doi = "10.1111/emip.12296",
language = "English (US)",
journal = "Educational Measurement: Issues and Practice",
issn = "0731-1745",
publisher = "Wiley-Blackwell",

}

TY - JOUR

T1 - Improving the Measurement of School Climate Using Item Response Theory

AU - Lindstrom Johnson, Sarah

AU - Reichenberg, Ray E.

AU - Shukla, Kathan

AU - Waasdorp, Tracy E.

AU - Bradshaw, Catherine P.

PY - 2019/1/1

Y1 - 2019/1/1

N2 - The U.S. government has become increasingly focused on school climate, as recently evidenced by its inclusion as an accountability indicator in the Every Student Succeeds Act. Yet, there remains considerable variability in both conceptualizing and measuring school climate. To better inform the research and practice related to school climate and its measurement, we leveraged item response theory (IRT), a commonly used psychometric approach for the design of achievement assessments, to create a parsimonious measure of school climate that operates across varying individual characteristics. Students (n = 69,513) in 111 secondary schools completed a school climate assessment focused on three domains of climate (i.e., safety, engagement, and environment), as defined by the U.S. Department of Education. Item and test characteristics were estimated using the mirt package in R using unidimensional IRT. Analyses revealed measurement difficulties that resulted in a greater ability to assess less favorable perspectives on school climate. Differential item functioning analyses indicated measurement differences based on student academic success. These findings support the development of a broad measure of school climate but also highlight the importance of work to ensure precision in measuring school climate, particularly when considering use as an accountability measure.

AB - The U.S. government has become increasingly focused on school climate, as recently evidenced by its inclusion as an accountability indicator in the Every Student Succeeds Act. Yet, there remains considerable variability in both conceptualizing and measuring school climate. To better inform the research and practice related to school climate and its measurement, we leveraged item response theory (IRT), a commonly used psychometric approach for the design of achievement assessments, to create a parsimonious measure of school climate that operates across varying individual characteristics. Students (n = 69,513) in 111 secondary schools completed a school climate assessment focused on three domains of climate (i.e., safety, engagement, and environment), as defined by the U.S. Department of Education. Item and test characteristics were estimated using the mirt package in R using unidimensional IRT. Analyses revealed measurement difficulties that resulted in a greater ability to assess less favorable perspectives on school climate. Differential item functioning analyses indicated measurement differences based on student academic success. These findings support the development of a broad measure of school climate but also highlight the importance of work to ensure precision in measuring school climate, particularly when considering use as an accountability measure.

KW - item response theory

KW - measurement invariance

KW - school accountability

KW - school climate

UR - http://www.scopus.com/inward/record.url?scp=85074041434&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85074041434&partnerID=8YFLogxK

U2 - 10.1111/emip.12296

DO - 10.1111/emip.12296

M3 - Article

AN - SCOPUS:85074041434

JO - Educational Measurement: Issues and Practice

JF - Educational Measurement: Issues and Practice

SN - 0731-1745

ER -