Applications of text analysis tools for spoken response grading

Scott Crossley, Danielle McNamara

Research output: Contribution to journalArticle

20 Citations (Scopus)

Abstract

This study explores the potential for automated indices related to speech delivery, language use, and topic development to model human judgments of TOEFL speaking proficiency in second language (L2) speech samples. For this study, 244 transcribed TOEFL speech samples taken from 244 L2 learners were analyzed using automated indices taken from Coh-Metrix, CPIDR, and LIWC. A stepwise linear regression was used to explain the variance in human judgments of independent speaking ability and overall speaking proficiency. Automated indices related to word type counts, causal cohesion, and lexical diversity predicted 52%of the variance in human ratings for the independent speech samples. Automated indices related to word type counts and word frequency predicted 61% of the variance of the human scores of overall speaking proficiency. These analyses demonstrate that, even in the absence of indices related to pronunciation and prosody (e.g., phonological accuracy, intonation, and stress), automated indices related to vocabulary size, causality, and word frequency can predict a significant amount of the variance in human ratings of speaking proficiency. These findings have important implications for understanding the construct of speaking proficiency and for the development of automatic scoring techniques.

Original languageEnglish (US)
Pages (from-to)171-192
Number of pages22
JournalLanguage Learning and Technology
Volume17
Issue number2
StatePublished - Jun 2013

Fingerprint

text analysis
grading
speaking
rating
Linear regression
language
causality
Grading
Text Analysis
vocabulary
Proficiency
regression
ability

Keywords

  • Computational linguistics
  • Corpus linguistics
  • Language testing
  • Machine learning
  • Speaking proficiency

ASJC Scopus subject areas

  • Education
  • Linguistics and Language
  • Computer Science Applications
  • Language and Linguistics

Cite this

Applications of text analysis tools for spoken response grading. / Crossley, Scott; McNamara, Danielle.

In: Language Learning and Technology, Vol. 17, No. 2, 06.2013, p. 171-192.

Research output: Contribution to journalArticle

@article{7f6f67529104467b80aee9711a4ed3b7,
title = "Applications of text analysis tools for spoken response grading",
abstract = "This study explores the potential for automated indices related to speech delivery, language use, and topic development to model human judgments of TOEFL speaking proficiency in second language (L2) speech samples. For this study, 244 transcribed TOEFL speech samples taken from 244 L2 learners were analyzed using automated indices taken from Coh-Metrix, CPIDR, and LIWC. A stepwise linear regression was used to explain the variance in human judgments of independent speaking ability and overall speaking proficiency. Automated indices related to word type counts, causal cohesion, and lexical diversity predicted 52{\%}of the variance in human ratings for the independent speech samples. Automated indices related to word type counts and word frequency predicted 61{\%} of the variance of the human scores of overall speaking proficiency. These analyses demonstrate that, even in the absence of indices related to pronunciation and prosody (e.g., phonological accuracy, intonation, and stress), automated indices related to vocabulary size, causality, and word frequency can predict a significant amount of the variance in human ratings of speaking proficiency. These findings have important implications for understanding the construct of speaking proficiency and for the development of automatic scoring techniques.",
keywords = "Computational linguistics, Corpus linguistics, Language testing, Machine learning, Speaking proficiency",
author = "Scott Crossley and Danielle McNamara",
year = "2013",
month = "6",
language = "English (US)",
volume = "17",
pages = "171--192",
journal = "Language Learning and Technology",
issn = "1094-3501",
publisher = "University of Hawaii Press",
number = "2",

}

TY - JOUR

T1 - Applications of text analysis tools for spoken response grading

AU - Crossley, Scott

AU - McNamara, Danielle

PY - 2013/6

Y1 - 2013/6

N2 - This study explores the potential for automated indices related to speech delivery, language use, and topic development to model human judgments of TOEFL speaking proficiency in second language (L2) speech samples. For this study, 244 transcribed TOEFL speech samples taken from 244 L2 learners were analyzed using automated indices taken from Coh-Metrix, CPIDR, and LIWC. A stepwise linear regression was used to explain the variance in human judgments of independent speaking ability and overall speaking proficiency. Automated indices related to word type counts, causal cohesion, and lexical diversity predicted 52%of the variance in human ratings for the independent speech samples. Automated indices related to word type counts and word frequency predicted 61% of the variance of the human scores of overall speaking proficiency. These analyses demonstrate that, even in the absence of indices related to pronunciation and prosody (e.g., phonological accuracy, intonation, and stress), automated indices related to vocabulary size, causality, and word frequency can predict a significant amount of the variance in human ratings of speaking proficiency. These findings have important implications for understanding the construct of speaking proficiency and for the development of automatic scoring techniques.

AB - This study explores the potential for automated indices related to speech delivery, language use, and topic development to model human judgments of TOEFL speaking proficiency in second language (L2) speech samples. For this study, 244 transcribed TOEFL speech samples taken from 244 L2 learners were analyzed using automated indices taken from Coh-Metrix, CPIDR, and LIWC. A stepwise linear regression was used to explain the variance in human judgments of independent speaking ability and overall speaking proficiency. Automated indices related to word type counts, causal cohesion, and lexical diversity predicted 52%of the variance in human ratings for the independent speech samples. Automated indices related to word type counts and word frequency predicted 61% of the variance of the human scores of overall speaking proficiency. These analyses demonstrate that, even in the absence of indices related to pronunciation and prosody (e.g., phonological accuracy, intonation, and stress), automated indices related to vocabulary size, causality, and word frequency can predict a significant amount of the variance in human ratings of speaking proficiency. These findings have important implications for understanding the construct of speaking proficiency and for the development of automatic scoring techniques.

KW - Computational linguistics

KW - Corpus linguistics

KW - Language testing

KW - Machine learning

KW - Speaking proficiency

UR - http://www.scopus.com/inward/record.url?scp=84878513315&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84878513315&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:84878513315

VL - 17

SP - 171

EP - 192

JO - Language Learning and Technology

JF - Language Learning and Technology

SN - 1094-3501

IS - 2

ER -