Natural language processing in an intelligent writing strategy tutoring system

Danielle McNamara, Scott A. Crossley, Rod Roscoe

Research output: Contribution to journalArticle

59 Citations (Scopus)

Abstract

The Writing Pal is an intelligent tutoring system that provides writing strategy training. A large part of its artificial intelligence resides in the natural language processing algorithms to assess essay quality and guide feedback to students. Because writing is often highly nuanced and subjective, the development of these algorithms must consider a broad array of linguistic, rhetorical, and contextual features. This study assesses the potential for computational indices to predict human ratings of essay quality. Past studies have demonstrated that linguistic indices related to lexical diversity, word frequency, and syntactic complexity are significant predictors of human judgments of essay quality but that indices of cohesion are not. The present study extends prior work by including a larger data sample and an expanded set of indices to assess new lexical, syntactic, cohesion, rhetorical, and reading ease indices. Three models were assessed. The model reported by McNamara, Crossley, and McCarthy (Written Communication 27:57-86, 2010) including three indices of lexical diversity, word frequency, and syntactic complexity accounted for only 6 % of the variance in the larger data set. A regression model including the full set of indices examined in prior studies of writing predicted 38 % of the variance in human scores of essay quality with 91 % adjacent accuracy (i.e., within 1 point). A regression model that also included new indices related to rhetoric and cohesion predicted 44 % of the variance with 94 % adjacent accuracy. The new indices increased accuracy but, more importantly, afford the means to provide more meaningful feedback in the context of a writing tutoring system.

Original languageEnglish (US)
Pages (from-to)499-515
Number of pages17
JournalBehavior Research Methods
Volume45
Issue number2
DOIs
StatePublished - Jun 2013

Fingerprint

Natural Language Processing
Linguistics
Artificial Intelligence
Reading
Communication
Writing Strategies
Tutoring
Students

Keywords

  • Automated essay scoring
  • Computational linguistics
  • Corpus linguistics
  • Intelligent tutoring systems
  • Natural language processing
  • Writing pedagogy

ASJC Scopus subject areas

  • Psychology(all)
  • Psychology (miscellaneous)
  • Experimental and Cognitive Psychology
  • Arts and Humanities (miscellaneous)
  • Developmental and Educational Psychology

Cite this

Natural language processing in an intelligent writing strategy tutoring system. / McNamara, Danielle; Crossley, Scott A.; Roscoe, Rod.

In: Behavior Research Methods, Vol. 45, No. 2, 06.2013, p. 499-515.

Research output: Contribution to journalArticle

@article{d13573887f53478888bae7f0d42a406d,
title = "Natural language processing in an intelligent writing strategy tutoring system",
abstract = "The Writing Pal is an intelligent tutoring system that provides writing strategy training. A large part of its artificial intelligence resides in the natural language processing algorithms to assess essay quality and guide feedback to students. Because writing is often highly nuanced and subjective, the development of these algorithms must consider a broad array of linguistic, rhetorical, and contextual features. This study assesses the potential for computational indices to predict human ratings of essay quality. Past studies have demonstrated that linguistic indices related to lexical diversity, word frequency, and syntactic complexity are significant predictors of human judgments of essay quality but that indices of cohesion are not. The present study extends prior work by including a larger data sample and an expanded set of indices to assess new lexical, syntactic, cohesion, rhetorical, and reading ease indices. Three models were assessed. The model reported by McNamara, Crossley, and McCarthy (Written Communication 27:57-86, 2010) including three indices of lexical diversity, word frequency, and syntactic complexity accounted for only 6 {\%} of the variance in the larger data set. A regression model including the full set of indices examined in prior studies of writing predicted 38 {\%} of the variance in human scores of essay quality with 91 {\%} adjacent accuracy (i.e., within 1 point). A regression model that also included new indices related to rhetoric and cohesion predicted 44 {\%} of the variance with 94 {\%} adjacent accuracy. The new indices increased accuracy but, more importantly, afford the means to provide more meaningful feedback in the context of a writing tutoring system.",
keywords = "Automated essay scoring, Computational linguistics, Corpus linguistics, Intelligent tutoring systems, Natural language processing, Writing pedagogy",
author = "Danielle McNamara and Crossley, {Scott A.} and Rod Roscoe",
year = "2013",
month = "6",
doi = "10.3758/s13428-012-0258-1",
language = "English (US)",
volume = "45",
pages = "499--515",
journal = "Behavior Research Methods, Instruments, and Computers",
issn = "1554-351X",
publisher = "Springer New York",
number = "2",

}

TY - JOUR

T1 - Natural language processing in an intelligent writing strategy tutoring system

AU - McNamara, Danielle

AU - Crossley, Scott A.

AU - Roscoe, Rod

PY - 2013/6

Y1 - 2013/6

N2 - The Writing Pal is an intelligent tutoring system that provides writing strategy training. A large part of its artificial intelligence resides in the natural language processing algorithms to assess essay quality and guide feedback to students. Because writing is often highly nuanced and subjective, the development of these algorithms must consider a broad array of linguistic, rhetorical, and contextual features. This study assesses the potential for computational indices to predict human ratings of essay quality. Past studies have demonstrated that linguistic indices related to lexical diversity, word frequency, and syntactic complexity are significant predictors of human judgments of essay quality but that indices of cohesion are not. The present study extends prior work by including a larger data sample and an expanded set of indices to assess new lexical, syntactic, cohesion, rhetorical, and reading ease indices. Three models were assessed. The model reported by McNamara, Crossley, and McCarthy (Written Communication 27:57-86, 2010) including three indices of lexical diversity, word frequency, and syntactic complexity accounted for only 6 % of the variance in the larger data set. A regression model including the full set of indices examined in prior studies of writing predicted 38 % of the variance in human scores of essay quality with 91 % adjacent accuracy (i.e., within 1 point). A regression model that also included new indices related to rhetoric and cohesion predicted 44 % of the variance with 94 % adjacent accuracy. The new indices increased accuracy but, more importantly, afford the means to provide more meaningful feedback in the context of a writing tutoring system.

AB - The Writing Pal is an intelligent tutoring system that provides writing strategy training. A large part of its artificial intelligence resides in the natural language processing algorithms to assess essay quality and guide feedback to students. Because writing is often highly nuanced and subjective, the development of these algorithms must consider a broad array of linguistic, rhetorical, and contextual features. This study assesses the potential for computational indices to predict human ratings of essay quality. Past studies have demonstrated that linguistic indices related to lexical diversity, word frequency, and syntactic complexity are significant predictors of human judgments of essay quality but that indices of cohesion are not. The present study extends prior work by including a larger data sample and an expanded set of indices to assess new lexical, syntactic, cohesion, rhetorical, and reading ease indices. Three models were assessed. The model reported by McNamara, Crossley, and McCarthy (Written Communication 27:57-86, 2010) including three indices of lexical diversity, word frequency, and syntactic complexity accounted for only 6 % of the variance in the larger data set. A regression model including the full set of indices examined in prior studies of writing predicted 38 % of the variance in human scores of essay quality with 91 % adjacent accuracy (i.e., within 1 point). A regression model that also included new indices related to rhetoric and cohesion predicted 44 % of the variance with 94 % adjacent accuracy. The new indices increased accuracy but, more importantly, afford the means to provide more meaningful feedback in the context of a writing tutoring system.

KW - Automated essay scoring

KW - Computational linguistics

KW - Corpus linguistics

KW - Intelligent tutoring systems

KW - Natural language processing

KW - Writing pedagogy

UR - http://www.scopus.com/inward/record.url?scp=84878218112&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84878218112&partnerID=8YFLogxK

U2 - 10.3758/s13428-012-0258-1

DO - 10.3758/s13428-012-0258-1

M3 - Article

C2 - 23055164

AN - SCOPUS:84878218112

VL - 45

SP - 499

EP - 515

JO - Behavior Research Methods, Instruments, and Computers

JF - Behavior Research Methods, Instruments, and Computers

SN - 1554-351X

IS - 2

ER -