A review of three large-scale datasets critiquing item design, data collection, and the usefulness of claims

Darryl Orletsky, James Middleton, Finbarr Sloane

Research output: Chapter in Book/Report/Conference proceedingChapter

1 Citation (Scopus)

Abstract

Issues of validity and usefulness of three large-scale longitudinal data sets are reviewed in this chapter. The Trends in International Mathematics and Science Study (TIMSS), the National Assessment of Educational Progress (NAEP), and the Educational Longitudinal Study of 2002 (ELS:2002) are compared and contrasted with respect to differences in sampling frame, internal and external validity, and especially construct validity of assessment items. Conclusions about the usefulness of large-scale secondary data analysis show that the reviewed assessments have been critical for determining inequities of opportunity for gender, ethnicity, socioeconomic status, and across national boundaries. They have also been useful for researchers examining the effectiveness of curricular policy on student learning. Moreover, some stakeholders have used the results as evidence that a nation’s future GDP is predicted by the outcome on TIMSS, and that students need more mathematical knowledge and skills to compete in a world that has an ever increasing rate of technological expansion. Though longitudinal, the duration of the studies presents a problem, as none follow students’ mathematical abilities or development for any length of time (e.g., early childhood into adulthood), and few studies from large-scale assessments shed light onto the kinds of pedagogy or curricular tasks that positively impact student learning. Lastly, threats to validity for large-scale studies are critiqued, and shown to be underreported in the literature.

Original languageEnglish (US)
Title of host publicationLarge-Scale Studies in Mathematics Education
PublisherSpringer International Publishing
Pages311-334
Number of pages24
ISBN (Print)9783319077161, 9783319077154
DOIs
StatePublished - Jan 1 2015

Fingerprint

mathematics studies
science studies
student
secondary analysis
trend
construct validity
adulthood
learning
social status
longitudinal study
data analysis
ethnicity
childhood
stakeholder
threat
gender
ability
evidence
time
literature

ASJC Scopus subject areas

  • Social Sciences(all)

Cite this

Orletsky, D., Middleton, J., & Sloane, F. (2015). A review of three large-scale datasets critiquing item design, data collection, and the usefulness of claims. In Large-Scale Studies in Mathematics Education (pp. 311-334). Springer International Publishing. https://doi.org/10.1007/978-3-319-07716-1_14

A review of three large-scale datasets critiquing item design, data collection, and the usefulness of claims. / Orletsky, Darryl; Middleton, James; Sloane, Finbarr.

Large-Scale Studies in Mathematics Education. Springer International Publishing, 2015. p. 311-334.

Research output: Chapter in Book/Report/Conference proceedingChapter

Orletsky, D, Middleton, J & Sloane, F 2015, A review of three large-scale datasets critiquing item design, data collection, and the usefulness of claims. in Large-Scale Studies in Mathematics Education. Springer International Publishing, pp. 311-334. https://doi.org/10.1007/978-3-319-07716-1_14
Orletsky D, Middleton J, Sloane F. A review of three large-scale datasets critiquing item design, data collection, and the usefulness of claims. In Large-Scale Studies in Mathematics Education. Springer International Publishing. 2015. p. 311-334 https://doi.org/10.1007/978-3-319-07716-1_14
Orletsky, Darryl ; Middleton, James ; Sloane, Finbarr. / A review of three large-scale datasets critiquing item design, data collection, and the usefulness of claims. Large-Scale Studies in Mathematics Education. Springer International Publishing, 2015. pp. 311-334
@inbook{e9882f29e2164a1d8e14f17fcfbfca41,
title = "A review of three large-scale datasets critiquing item design, data collection, and the usefulness of claims",
abstract = "Issues of validity and usefulness of three large-scale longitudinal data sets are reviewed in this chapter. The Trends in International Mathematics and Science Study (TIMSS), the National Assessment of Educational Progress (NAEP), and the Educational Longitudinal Study of 2002 (ELS:2002) are compared and contrasted with respect to differences in sampling frame, internal and external validity, and especially construct validity of assessment items. Conclusions about the usefulness of large-scale secondary data analysis show that the reviewed assessments have been critical for determining inequities of opportunity for gender, ethnicity, socioeconomic status, and across national boundaries. They have also been useful for researchers examining the effectiveness of curricular policy on student learning. Moreover, some stakeholders have used the results as evidence that a nation’s future GDP is predicted by the outcome on TIMSS, and that students need more mathematical knowledge and skills to compete in a world that has an ever increasing rate of technological expansion. Though longitudinal, the duration of the studies presents a problem, as none follow students’ mathematical abilities or development for any length of time (e.g., early childhood into adulthood), and few studies from large-scale assessments shed light onto the kinds of pedagogy or curricular tasks that positively impact student learning. Lastly, threats to validity for large-scale studies are critiqued, and shown to be underreported in the literature.",
author = "Darryl Orletsky and James Middleton and Finbarr Sloane",
year = "2015",
month = "1",
day = "1",
doi = "10.1007/978-3-319-07716-1_14",
language = "English (US)",
isbn = "9783319077161",
pages = "311--334",
booktitle = "Large-Scale Studies in Mathematics Education",
publisher = "Springer International Publishing",

}

TY - CHAP

T1 - A review of three large-scale datasets critiquing item design, data collection, and the usefulness of claims

AU - Orletsky, Darryl

AU - Middleton, James

AU - Sloane, Finbarr

PY - 2015/1/1

Y1 - 2015/1/1

N2 - Issues of validity and usefulness of three large-scale longitudinal data sets are reviewed in this chapter. The Trends in International Mathematics and Science Study (TIMSS), the National Assessment of Educational Progress (NAEP), and the Educational Longitudinal Study of 2002 (ELS:2002) are compared and contrasted with respect to differences in sampling frame, internal and external validity, and especially construct validity of assessment items. Conclusions about the usefulness of large-scale secondary data analysis show that the reviewed assessments have been critical for determining inequities of opportunity for gender, ethnicity, socioeconomic status, and across national boundaries. They have also been useful for researchers examining the effectiveness of curricular policy on student learning. Moreover, some stakeholders have used the results as evidence that a nation’s future GDP is predicted by the outcome on TIMSS, and that students need more mathematical knowledge and skills to compete in a world that has an ever increasing rate of technological expansion. Though longitudinal, the duration of the studies presents a problem, as none follow students’ mathematical abilities or development for any length of time (e.g., early childhood into adulthood), and few studies from large-scale assessments shed light onto the kinds of pedagogy or curricular tasks that positively impact student learning. Lastly, threats to validity for large-scale studies are critiqued, and shown to be underreported in the literature.

AB - Issues of validity and usefulness of three large-scale longitudinal data sets are reviewed in this chapter. The Trends in International Mathematics and Science Study (TIMSS), the National Assessment of Educational Progress (NAEP), and the Educational Longitudinal Study of 2002 (ELS:2002) are compared and contrasted with respect to differences in sampling frame, internal and external validity, and especially construct validity of assessment items. Conclusions about the usefulness of large-scale secondary data analysis show that the reviewed assessments have been critical for determining inequities of opportunity for gender, ethnicity, socioeconomic status, and across national boundaries. They have also been useful for researchers examining the effectiveness of curricular policy on student learning. Moreover, some stakeholders have used the results as evidence that a nation’s future GDP is predicted by the outcome on TIMSS, and that students need more mathematical knowledge and skills to compete in a world that has an ever increasing rate of technological expansion. Though longitudinal, the duration of the studies presents a problem, as none follow students’ mathematical abilities or development for any length of time (e.g., early childhood into adulthood), and few studies from large-scale assessments shed light onto the kinds of pedagogy or curricular tasks that positively impact student learning. Lastly, threats to validity for large-scale studies are critiqued, and shown to be underreported in the literature.

UR - http://www.scopus.com/inward/record.url?scp=84944565183&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84944565183&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-07716-1_14

DO - 10.1007/978-3-319-07716-1_14

M3 - Chapter

AN - SCOPUS:84944565183

SN - 9783319077161

SN - 9783319077154

SP - 311

EP - 334

BT - Large-Scale Studies in Mathematics Education

PB - Springer International Publishing

ER -