Different teacher-level effectiveness estimates, different results

inter-model concordance across six generalized value-added models (VAMs)

Edward Sloat, Audrey Beardsley, Jessica Holloway

Research output: Contribution to journalArticle

Abstract

In this study, researchers compared the concordance of teacher-level effectiveness ratings derived via six common generalized value-added model (VAM) approaches including a (1) student growth percentile (SGP) model, (2) value-added linear regression model (VALRM), (3) value-added hierarchical linear model (VAHLM), (4) simple difference (gain) score model, (5) rubric-based performance level (growth) model, and (6) simple criterion (percent passing) model. The study sample included fourth to sixth grade teachers employed in a large, suburban school district who taught the same sets of students, at the same time, and for whom a consistent set of achievement measures and background variables were available. Findings indicate that ratings significantly and substantively differed depending upon the methodological approach used. Findings, accordingly, bring into question the validity of the inferences based on such estimates, especially when high-stakes decisions are made about teachers as based on estimates measured via different, albeit popular methods across different school districts and states.

Original languageEnglish (US)
JournalEducational Assessment, Evaluation and Accountability
DOIs
StateAccepted/In press - Jan 1 2018

Fingerprint

value added
teacher
rating
district
linear model
school
Value added
Concordance
student
school grade
regression
Growth model
School districts
Rating
performance

Keywords

  • Teacher accountability
  • Teacher effectiveness
  • Teacher evaluation
  • Teacher quality
  • Validity
  • Value-added models

ASJC Scopus subject areas

  • Education
  • Organizational Behavior and Human Resource Management

Cite this

@article{2d6e57930b76465d971cf8bf55ff09fc,
title = "Different teacher-level effectiveness estimates, different results: inter-model concordance across six generalized value-added models (VAMs)",
abstract = "In this study, researchers compared the concordance of teacher-level effectiveness ratings derived via six common generalized value-added model (VAM) approaches including a (1) student growth percentile (SGP) model, (2) value-added linear regression model (VALRM), (3) value-added hierarchical linear model (VAHLM), (4) simple difference (gain) score model, (5) rubric-based performance level (growth) model, and (6) simple criterion (percent passing) model. The study sample included fourth to sixth grade teachers employed in a large, suburban school district who taught the same sets of students, at the same time, and for whom a consistent set of achievement measures and background variables were available. Findings indicate that ratings significantly and substantively differed depending upon the methodological approach used. Findings, accordingly, bring into question the validity of the inferences based on such estimates, especially when high-stakes decisions are made about teachers as based on estimates measured via different, albeit popular methods across different school districts and states.",
keywords = "Teacher accountability, Teacher effectiveness, Teacher evaluation, Teacher quality, Validity, Value-added models",
author = "Edward Sloat and Audrey Beardsley and Jessica Holloway",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/s11092-018-9283-7",
language = "English (US)",
journal = "Educational Assessment, Evaluation and Accountability",
issn = "1874-8597",
publisher = "Springer Netherlands",

}

TY - JOUR

T1 - Different teacher-level effectiveness estimates, different results

T2 - inter-model concordance across six generalized value-added models (VAMs)

AU - Sloat, Edward

AU - Beardsley, Audrey

AU - Holloway, Jessica

PY - 2018/1/1

Y1 - 2018/1/1

N2 - In this study, researchers compared the concordance of teacher-level effectiveness ratings derived via six common generalized value-added model (VAM) approaches including a (1) student growth percentile (SGP) model, (2) value-added linear regression model (VALRM), (3) value-added hierarchical linear model (VAHLM), (4) simple difference (gain) score model, (5) rubric-based performance level (growth) model, and (6) simple criterion (percent passing) model. The study sample included fourth to sixth grade teachers employed in a large, suburban school district who taught the same sets of students, at the same time, and for whom a consistent set of achievement measures and background variables were available. Findings indicate that ratings significantly and substantively differed depending upon the methodological approach used. Findings, accordingly, bring into question the validity of the inferences based on such estimates, especially when high-stakes decisions are made about teachers as based on estimates measured via different, albeit popular methods across different school districts and states.

AB - In this study, researchers compared the concordance of teacher-level effectiveness ratings derived via six common generalized value-added model (VAM) approaches including a (1) student growth percentile (SGP) model, (2) value-added linear regression model (VALRM), (3) value-added hierarchical linear model (VAHLM), (4) simple difference (gain) score model, (5) rubric-based performance level (growth) model, and (6) simple criterion (percent passing) model. The study sample included fourth to sixth grade teachers employed in a large, suburban school district who taught the same sets of students, at the same time, and for whom a consistent set of achievement measures and background variables were available. Findings indicate that ratings significantly and substantively differed depending upon the methodological approach used. Findings, accordingly, bring into question the validity of the inferences based on such estimates, especially when high-stakes decisions are made about teachers as based on estimates measured via different, albeit popular methods across different school districts and states.

KW - Teacher accountability

KW - Teacher effectiveness

KW - Teacher evaluation

KW - Teacher quality

KW - Validity

KW - Value-added models

UR - http://www.scopus.com/inward/record.url?scp=85050908415&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050908415&partnerID=8YFLogxK

U2 - 10.1007/s11092-018-9283-7

DO - 10.1007/s11092-018-9283-7

M3 - Article

JO - Educational Assessment, Evaluation and Accountability

JF - Educational Assessment, Evaluation and Accountability

SN - 1874-8597

ER -