Paragraph specific n-gram approaches to automatically assessing essay quality

Scott Crossley, Caleb DeFore, Kris Kyle, Jianmin Dai, Danielle McNamara

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we describe an n-gram approach to automatically assess essay quality in student writing. Underlying this approach is the development of n-gram indices that examine rhetorical, syntactic, grammatical, and cohesion features of paragraph types (introduction, body, and conclusion paragraphs) and entire essays. For this study, we developed over 300 n-gram indices and assessed their potential to predict human ratings of essay quality. A combination of these n-gram indices explained over 30% of the variance in human ratings for essays in a training and testing corpus. The findings from this study indicate the strength of using n-gram indices to automatically assess writing quality. Such indices not only explain text-based factors that influence human judgments of essay quality, but also provide new methods for automatically assessing writing quality.

Original languageEnglish (US)
Title of host publicationProceedings of the 6th International Conference on Educational Data Mining, EDM 2013
EditorsSidney K. D'Mello, Rafael A. Calvo, Andrew Olney
PublisherInternational Educational Data Mining Society
ISBN (Electronic)9780983952527
StatePublished - Jan 1 2013
Event6th International Conference on Educational Data Mining, EDM 2013 - Memphis, United States
Duration: Jul 6 2013Jul 9 2013

Publication series

NameProceedings of the 6th International Conference on Educational Data Mining, EDM 2013

Conference

Conference6th International Conference on Educational Data Mining, EDM 2013
CountryUnited States
CityMemphis
Period7/6/137/9/13

Fingerprint

Syntactics
Students
Testing

Keywords

  • Automatic feedback
  • Computational linguistics
  • Corpus linguistics
  • Essay quality
  • Intelligent tutoring systems

ASJC Scopus subject areas

  • Computer Science Applications
  • Information Systems

Cite this

Crossley, S., DeFore, C., Kyle, K., Dai, J., & McNamara, D. (2013). Paragraph specific n-gram approaches to automatically assessing essay quality. In S. K. D'Mello, R. A. Calvo, & A. Olney (Eds.), Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013 (Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013). International Educational Data Mining Society.

Paragraph specific n-gram approaches to automatically assessing essay quality. / Crossley, Scott; DeFore, Caleb; Kyle, Kris; Dai, Jianmin; McNamara, Danielle.

Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013. ed. / Sidney K. D'Mello; Rafael A. Calvo; Andrew Olney. International Educational Data Mining Society, 2013. (Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Crossley, S, DeFore, C, Kyle, K, Dai, J & McNamara, D 2013, Paragraph specific n-gram approaches to automatically assessing essay quality. in SK D'Mello, RA Calvo & A Olney (eds), Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013. Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013, International Educational Data Mining Society, 6th International Conference on Educational Data Mining, EDM 2013, Memphis, United States, 7/6/13.
Crossley S, DeFore C, Kyle K, Dai J, McNamara D. Paragraph specific n-gram approaches to automatically assessing essay quality. In D'Mello SK, Calvo RA, Olney A, editors, Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013. International Educational Data Mining Society. 2013. (Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013).
Crossley, Scott ; DeFore, Caleb ; Kyle, Kris ; Dai, Jianmin ; McNamara, Danielle. / Paragraph specific n-gram approaches to automatically assessing essay quality. Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013. editor / Sidney K. D'Mello ; Rafael A. Calvo ; Andrew Olney. International Educational Data Mining Society, 2013. (Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013).
@inproceedings{ad0d65412ed04e8b82dde56daaaca781,
title = "Paragraph specific n-gram approaches to automatically assessing essay quality",
abstract = "In this paper, we describe an n-gram approach to automatically assess essay quality in student writing. Underlying this approach is the development of n-gram indices that examine rhetorical, syntactic, grammatical, and cohesion features of paragraph types (introduction, body, and conclusion paragraphs) and entire essays. For this study, we developed over 300 n-gram indices and assessed their potential to predict human ratings of essay quality. A combination of these n-gram indices explained over 30{\%} of the variance in human ratings for essays in a training and testing corpus. The findings from this study indicate the strength of using n-gram indices to automatically assess writing quality. Such indices not only explain text-based factors that influence human judgments of essay quality, but also provide new methods for automatically assessing writing quality.",
keywords = "Automatic feedback, Computational linguistics, Corpus linguistics, Essay quality, Intelligent tutoring systems",
author = "Scott Crossley and Caleb DeFore and Kris Kyle and Jianmin Dai and Danielle McNamara",
year = "2013",
month = "1",
day = "1",
language = "English (US)",
series = "Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013",
publisher = "International Educational Data Mining Society",
editor = "D'Mello, {Sidney K.} and Calvo, {Rafael A.} and Andrew Olney",
booktitle = "Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013",

}

TY - GEN

T1 - Paragraph specific n-gram approaches to automatically assessing essay quality

AU - Crossley, Scott

AU - DeFore, Caleb

AU - Kyle, Kris

AU - Dai, Jianmin

AU - McNamara, Danielle

PY - 2013/1/1

Y1 - 2013/1/1

N2 - In this paper, we describe an n-gram approach to automatically assess essay quality in student writing. Underlying this approach is the development of n-gram indices that examine rhetorical, syntactic, grammatical, and cohesion features of paragraph types (introduction, body, and conclusion paragraphs) and entire essays. For this study, we developed over 300 n-gram indices and assessed their potential to predict human ratings of essay quality. A combination of these n-gram indices explained over 30% of the variance in human ratings for essays in a training and testing corpus. The findings from this study indicate the strength of using n-gram indices to automatically assess writing quality. Such indices not only explain text-based factors that influence human judgments of essay quality, but also provide new methods for automatically assessing writing quality.

AB - In this paper, we describe an n-gram approach to automatically assess essay quality in student writing. Underlying this approach is the development of n-gram indices that examine rhetorical, syntactic, grammatical, and cohesion features of paragraph types (introduction, body, and conclusion paragraphs) and entire essays. For this study, we developed over 300 n-gram indices and assessed their potential to predict human ratings of essay quality. A combination of these n-gram indices explained over 30% of the variance in human ratings for essays in a training and testing corpus. The findings from this study indicate the strength of using n-gram indices to automatically assess writing quality. Such indices not only explain text-based factors that influence human judgments of essay quality, but also provide new methods for automatically assessing writing quality.

KW - Automatic feedback

KW - Computational linguistics

KW - Corpus linguistics

KW - Essay quality

KW - Intelligent tutoring systems

UR - http://www.scopus.com/inward/record.url?scp=84911912617&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84911912617&partnerID=8YFLogxK

M3 - Conference contribution

T3 - Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013

BT - Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013

A2 - D'Mello, Sidney K.

A2 - Calvo, Rafael A.

A2 - Olney, Andrew

PB - International Educational Data Mining Society

ER -