Automated Scoring of Students’ Small-Group Discussions to Assess Reading Ability

Audra E. Kosh, Jeffrey A. Greene, P. Karen Murphy, Hal Burdick, Carla Firetto, Jeff Elmore

Research output: Contribution to journalArticle

Abstract

We explored the feasibility of using automated scoring to assess upper-elementary students’ reading ability through analysis of transcripts of students’ small-group discussions about texts. Participants included 35 fourth-grade students across two classrooms that engaged in a literacy intervention called Quality Talk. During the course of one school year, data were collected at 10 time points for a total of 327 student-text encounters, with a different text discussed at each time point. To explore the possibility of automated scoring, we considered which quantitative discourse variables (e.g., variables to measure language sophistication and latent semantic analysis variables) were the strongest predictors of scores on a multiple-choice and constructed-response reading comprehension test. Convergent validity evidence was collected by comparing automatically calculated quantitative discourse features to scores on a reading fluency test. After examining a variety of discourse features using multilevel modeling, results showed that measures of word rareness and word diversity were the most promising variables to use in automated scoring of students’ discussions.

Original languageEnglish (US)
Pages (from-to)20-34
Number of pages15
JournalEducational Measurement: Issues and Practice
Volume37
Issue number2
DOIs
StatePublished - Jun 1 2018
Externally publishedYes

Fingerprint

group discussion
small group
ability
student
discourse
comprehension
literacy
semantics
classroom
language
school
evidence
time

Keywords

  • assessment
  • automated scoring
  • reading ability

ASJC Scopus subject areas

  • Education

Cite this

Automated Scoring of Students’ Small-Group Discussions to Assess Reading Ability. / Kosh, Audra E.; Greene, Jeffrey A.; Murphy, P. Karen; Burdick, Hal; Firetto, Carla; Elmore, Jeff.

In: Educational Measurement: Issues and Practice, Vol. 37, No. 2, 01.06.2018, p. 20-34.

Research output: Contribution to journalArticle

Kosh, Audra E. ; Greene, Jeffrey A. ; Murphy, P. Karen ; Burdick, Hal ; Firetto, Carla ; Elmore, Jeff. / Automated Scoring of Students’ Small-Group Discussions to Assess Reading Ability. In: Educational Measurement: Issues and Practice. 2018 ; Vol. 37, No. 2. pp. 20-34.
@article{63bd6a5636bc4ef585de5deed31dd1ee,
title = "Automated Scoring of Students’ Small-Group Discussions to Assess Reading Ability",
abstract = "We explored the feasibility of using automated scoring to assess upper-elementary students’ reading ability through analysis of transcripts of students’ small-group discussions about texts. Participants included 35 fourth-grade students across two classrooms that engaged in a literacy intervention called Quality Talk. During the course of one school year, data were collected at 10 time points for a total of 327 student-text encounters, with a different text discussed at each time point. To explore the possibility of automated scoring, we considered which quantitative discourse variables (e.g., variables to measure language sophistication and latent semantic analysis variables) were the strongest predictors of scores on a multiple-choice and constructed-response reading comprehension test. Convergent validity evidence was collected by comparing automatically calculated quantitative discourse features to scores on a reading fluency test. After examining a variety of discourse features using multilevel modeling, results showed that measures of word rareness and word diversity were the most promising variables to use in automated scoring of students’ discussions.",
keywords = "assessment, automated scoring, reading ability",
author = "Kosh, {Audra E.} and Greene, {Jeffrey A.} and Murphy, {P. Karen} and Hal Burdick and Carla Firetto and Jeff Elmore",
year = "2018",
month = "6",
day = "1",
doi = "10.1111/emip.12174",
language = "English (US)",
volume = "37",
pages = "20--34",
journal = "Educational Measurement: Issues and Practice",
issn = "0731-1745",
publisher = "Wiley-Blackwell",
number = "2",

}

TY - JOUR

T1 - Automated Scoring of Students’ Small-Group Discussions to Assess Reading Ability

AU - Kosh, Audra E.

AU - Greene, Jeffrey A.

AU - Murphy, P. Karen

AU - Burdick, Hal

AU - Firetto, Carla

AU - Elmore, Jeff

PY - 2018/6/1

Y1 - 2018/6/1

N2 - We explored the feasibility of using automated scoring to assess upper-elementary students’ reading ability through analysis of transcripts of students’ small-group discussions about texts. Participants included 35 fourth-grade students across two classrooms that engaged in a literacy intervention called Quality Talk. During the course of one school year, data were collected at 10 time points for a total of 327 student-text encounters, with a different text discussed at each time point. To explore the possibility of automated scoring, we considered which quantitative discourse variables (e.g., variables to measure language sophistication and latent semantic analysis variables) were the strongest predictors of scores on a multiple-choice and constructed-response reading comprehension test. Convergent validity evidence was collected by comparing automatically calculated quantitative discourse features to scores on a reading fluency test. After examining a variety of discourse features using multilevel modeling, results showed that measures of word rareness and word diversity were the most promising variables to use in automated scoring of students’ discussions.

AB - We explored the feasibility of using automated scoring to assess upper-elementary students’ reading ability through analysis of transcripts of students’ small-group discussions about texts. Participants included 35 fourth-grade students across two classrooms that engaged in a literacy intervention called Quality Talk. During the course of one school year, data were collected at 10 time points for a total of 327 student-text encounters, with a different text discussed at each time point. To explore the possibility of automated scoring, we considered which quantitative discourse variables (e.g., variables to measure language sophistication and latent semantic analysis variables) were the strongest predictors of scores on a multiple-choice and constructed-response reading comprehension test. Convergent validity evidence was collected by comparing automatically calculated quantitative discourse features to scores on a reading fluency test. After examining a variety of discourse features using multilevel modeling, results showed that measures of word rareness and word diversity were the most promising variables to use in automated scoring of students’ discussions.

KW - assessment

KW - automated scoring

KW - reading ability

UR - http://www.scopus.com/inward/record.url?scp=85034035294&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85034035294&partnerID=8YFLogxK

U2 - 10.1111/emip.12174

DO - 10.1111/emip.12174

M3 - Article

AN - SCOPUS:85034035294

VL - 37

SP - 20

EP - 34

JO - Educational Measurement: Issues and Practice

JF - Educational Measurement: Issues and Practice

SN - 0731-1745

IS - 2

ER -