Modified multiple-choice items for alternate assessments: Reliability, difficulty, and differential boost

Ryan J. Kettler, Michael C. Rodriguez, Daniel M. Bolt, Stephen Elliott, Peter A. Beddow, Alexander Kurz

Research output: Contribution to journalArticle

21 Citations (Scopus)

Abstract

Federal policy on alternate assessment based on modified academic achievement standards (AA-MAS) inspired this research. Specifically, an experimental study was conducted to determine whether tests composed of modified items would have the same level of reliability as tests composed of original items, and whether these modified items helped reduce the performance gap between AA-MAS eligible and ineligible students. Three groups of eighth-grade students (N = 755) defined by eligibility and disability status took original and modified versions of reading and mathematics tests. In a third condition, the students were provided limited reading support along with the modified items. Changes in reliability across groups and conditions for both the reading and mathematics tests were determined to be minimal. Mean item difficulties within the Rasch model were shown to decrease more for students who would be eligible for the AA-MAS than for non-eligible groups, revealing evidence of differential boost. Exploratory analyses indicated that shortening the question stem may be a highly effective modification, and that adding graphics to reading items may be a poor modification.

Original languageEnglish (US)
Pages (from-to)210-234
Number of pages25
JournalApplied Measurement in Education
Volume24
Issue number3
DOIs
StatePublished - Jul 2011

Fingerprint

Reading
academic achievement
Students
Mathematics
student
mathematics
federal policy
Group
disability
Research
performance
evidence

ASJC Scopus subject areas

  • Education
  • Developmental and Educational Psychology

Cite this

Modified multiple-choice items for alternate assessments : Reliability, difficulty, and differential boost. / Kettler, Ryan J.; Rodriguez, Michael C.; Bolt, Daniel M.; Elliott, Stephen; Beddow, Peter A.; Kurz, Alexander.

In: Applied Measurement in Education, Vol. 24, No. 3, 07.2011, p. 210-234.

Research output: Contribution to journalArticle

Kettler, Ryan J. ; Rodriguez, Michael C. ; Bolt, Daniel M. ; Elliott, Stephen ; Beddow, Peter A. ; Kurz, Alexander. / Modified multiple-choice items for alternate assessments : Reliability, difficulty, and differential boost. In: Applied Measurement in Education. 2011 ; Vol. 24, No. 3. pp. 210-234.
@article{9db1d2d72c284d70a259d03ba8cbe904,
title = "Modified multiple-choice items for alternate assessments: Reliability, difficulty, and differential boost",
abstract = "Federal policy on alternate assessment based on modified academic achievement standards (AA-MAS) inspired this research. Specifically, an experimental study was conducted to determine whether tests composed of modified items would have the same level of reliability as tests composed of original items, and whether these modified items helped reduce the performance gap between AA-MAS eligible and ineligible students. Three groups of eighth-grade students (N = 755) defined by eligibility and disability status took original and modified versions of reading and mathematics tests. In a third condition, the students were provided limited reading support along with the modified items. Changes in reliability across groups and conditions for both the reading and mathematics tests were determined to be minimal. Mean item difficulties within the Rasch model were shown to decrease more for students who would be eligible for the AA-MAS than for non-eligible groups, revealing evidence of differential boost. Exploratory analyses indicated that shortening the question stem may be a highly effective modification, and that adding graphics to reading items may be a poor modification.",
author = "Kettler, {Ryan J.} and Rodriguez, {Michael C.} and Bolt, {Daniel M.} and Stephen Elliott and Beddow, {Peter A.} and Alexander Kurz",
year = "2011",
month = "7",
doi = "10.1080/08957347.2011.580620",
language = "English (US)",
volume = "24",
pages = "210--234",
journal = "Applied Measurement in Education",
issn = "0895-7347",
publisher = "Routledge",
number = "3",

}

TY - JOUR

T1 - Modified multiple-choice items for alternate assessments

T2 - Reliability, difficulty, and differential boost

AU - Kettler, Ryan J.

AU - Rodriguez, Michael C.

AU - Bolt, Daniel M.

AU - Elliott, Stephen

AU - Beddow, Peter A.

AU - Kurz, Alexander

PY - 2011/7

Y1 - 2011/7

N2 - Federal policy on alternate assessment based on modified academic achievement standards (AA-MAS) inspired this research. Specifically, an experimental study was conducted to determine whether tests composed of modified items would have the same level of reliability as tests composed of original items, and whether these modified items helped reduce the performance gap between AA-MAS eligible and ineligible students. Three groups of eighth-grade students (N = 755) defined by eligibility and disability status took original and modified versions of reading and mathematics tests. In a third condition, the students were provided limited reading support along with the modified items. Changes in reliability across groups and conditions for both the reading and mathematics tests were determined to be minimal. Mean item difficulties within the Rasch model were shown to decrease more for students who would be eligible for the AA-MAS than for non-eligible groups, revealing evidence of differential boost. Exploratory analyses indicated that shortening the question stem may be a highly effective modification, and that adding graphics to reading items may be a poor modification.

AB - Federal policy on alternate assessment based on modified academic achievement standards (AA-MAS) inspired this research. Specifically, an experimental study was conducted to determine whether tests composed of modified items would have the same level of reliability as tests composed of original items, and whether these modified items helped reduce the performance gap between AA-MAS eligible and ineligible students. Three groups of eighth-grade students (N = 755) defined by eligibility and disability status took original and modified versions of reading and mathematics tests. In a third condition, the students were provided limited reading support along with the modified items. Changes in reliability across groups and conditions for both the reading and mathematics tests were determined to be minimal. Mean item difficulties within the Rasch model were shown to decrease more for students who would be eligible for the AA-MAS than for non-eligible groups, revealing evidence of differential boost. Exploratory analyses indicated that shortening the question stem may be a highly effective modification, and that adding graphics to reading items may be a poor modification.

UR - http://www.scopus.com/inward/record.url?scp=79960186407&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79960186407&partnerID=8YFLogxK

U2 - 10.1080/08957347.2011.580620

DO - 10.1080/08957347.2011.580620

M3 - Article

AN - SCOPUS:79960186407

VL - 24

SP - 210

EP - 234

JO - Applied Measurement in Education

JF - Applied Measurement in Education

SN - 0895-7347

IS - 3

ER -