A hierarchical classification approach to automated essay scoring

Danielle McNamara, Scott A. Crossley, Rod Roscoe, Laura K. Allen, Jianmin Dai

Research output: Contribution to journalArticle

55 Citations (Scopus)

Abstract

This study evaluates the use of a hierarchical classification approach to automated assessment of essays. Automated essay scoring (AES) generally relies on machine learning techniques that compute essay scores using a set of text variables. Unlike previous studies that rely on regression models, this study computes essay scores using a hierarchical approach, analogous to an incremental algorithm for hierarchical classification. The corpus in this study consists of 1243 argumentative (persuasive) essays written on 14 different prompts, across 3 different grade levels (9th grade, 11th grade, college freshman), and four different time limits for writing or temporal conditions (untimed essays and essays written in 10, 15, and 25. minute increments). The features included in the analysis are computed using the automated tools, Coh-Metrix, the Writing Assessment Tool (WAT), and Linguistic Inquiry and Word Count (LIWC). Overall, the models developed to score all the essays in the data set report 55% exact accuracy and 92% adjacent accuracy between the predicted essay scores and the human scores. The results indicate that this is a promising approach to AES that could provide more specific feedback to writers and may be relevant to other natural language computations, such as the scoring of short answers in comprehension or knowledge assessments.

Original languageEnglish (US)
Pages (from-to)35-59
Number of pages25
JournalAssessing Writing
Volume23
DOIs
StatePublished - Jan 1 2015

Fingerprint

school grade
Scoring
comprehension
writer
linguistics
regression
language
learning
Assessment Tools
Freshman
Writing Assessment
Machine Learning
Linguistic Inquiry
Writer
Natural Language
Grade Level
Prompts
time

Keywords

  • AES
  • Automated essay scoring
  • Hierarchical classification
  • Writing assessment

ASJC Scopus subject areas

  • Language and Linguistics
  • Education
  • Linguistics and Language

Cite this

A hierarchical classification approach to automated essay scoring. / McNamara, Danielle; Crossley, Scott A.; Roscoe, Rod; Allen, Laura K.; Dai, Jianmin.

In: Assessing Writing, Vol. 23, 01.01.2015, p. 35-59.

Research output: Contribution to journalArticle

@article{895c37b1ee5b4ccea653b75be380d501,
title = "A hierarchical classification approach to automated essay scoring",
abstract = "This study evaluates the use of a hierarchical classification approach to automated assessment of essays. Automated essay scoring (AES) generally relies on machine learning techniques that compute essay scores using a set of text variables. Unlike previous studies that rely on regression models, this study computes essay scores using a hierarchical approach, analogous to an incremental algorithm for hierarchical classification. The corpus in this study consists of 1243 argumentative (persuasive) essays written on 14 different prompts, across 3 different grade levels (9th grade, 11th grade, college freshman), and four different time limits for writing or temporal conditions (untimed essays and essays written in 10, 15, and 25. minute increments). The features included in the analysis are computed using the automated tools, Coh-Metrix, the Writing Assessment Tool (WAT), and Linguistic Inquiry and Word Count (LIWC). Overall, the models developed to score all the essays in the data set report 55{\%} exact accuracy and 92{\%} adjacent accuracy between the predicted essay scores and the human scores. The results indicate that this is a promising approach to AES that could provide more specific feedback to writers and may be relevant to other natural language computations, such as the scoring of short answers in comprehension or knowledge assessments.",
keywords = "AES, Automated essay scoring, Hierarchical classification, Writing assessment",
author = "Danielle McNamara and Crossley, {Scott A.} and Rod Roscoe and Allen, {Laura K.} and Jianmin Dai",
year = "2015",
month = "1",
day = "1",
doi = "10.1016/j.asw.2014.09.002",
language = "English (US)",
volume = "23",
pages = "35--59",
journal = "Assessing Writing",
issn = "1075-2935",
publisher = "Elsevier Limited",

}

TY - JOUR

T1 - A hierarchical classification approach to automated essay scoring

AU - McNamara, Danielle

AU - Crossley, Scott A.

AU - Roscoe, Rod

AU - Allen, Laura K.

AU - Dai, Jianmin

PY - 2015/1/1

Y1 - 2015/1/1

N2 - This study evaluates the use of a hierarchical classification approach to automated assessment of essays. Automated essay scoring (AES) generally relies on machine learning techniques that compute essay scores using a set of text variables. Unlike previous studies that rely on regression models, this study computes essay scores using a hierarchical approach, analogous to an incremental algorithm for hierarchical classification. The corpus in this study consists of 1243 argumentative (persuasive) essays written on 14 different prompts, across 3 different grade levels (9th grade, 11th grade, college freshman), and four different time limits for writing or temporal conditions (untimed essays and essays written in 10, 15, and 25. minute increments). The features included in the analysis are computed using the automated tools, Coh-Metrix, the Writing Assessment Tool (WAT), and Linguistic Inquiry and Word Count (LIWC). Overall, the models developed to score all the essays in the data set report 55% exact accuracy and 92% adjacent accuracy between the predicted essay scores and the human scores. The results indicate that this is a promising approach to AES that could provide more specific feedback to writers and may be relevant to other natural language computations, such as the scoring of short answers in comprehension or knowledge assessments.

AB - This study evaluates the use of a hierarchical classification approach to automated assessment of essays. Automated essay scoring (AES) generally relies on machine learning techniques that compute essay scores using a set of text variables. Unlike previous studies that rely on regression models, this study computes essay scores using a hierarchical approach, analogous to an incremental algorithm for hierarchical classification. The corpus in this study consists of 1243 argumentative (persuasive) essays written on 14 different prompts, across 3 different grade levels (9th grade, 11th grade, college freshman), and four different time limits for writing or temporal conditions (untimed essays and essays written in 10, 15, and 25. minute increments). The features included in the analysis are computed using the automated tools, Coh-Metrix, the Writing Assessment Tool (WAT), and Linguistic Inquiry and Word Count (LIWC). Overall, the models developed to score all the essays in the data set report 55% exact accuracy and 92% adjacent accuracy between the predicted essay scores and the human scores. The results indicate that this is a promising approach to AES that could provide more specific feedback to writers and may be relevant to other natural language computations, such as the scoring of short answers in comprehension or knowledge assessments.

KW - AES

KW - Automated essay scoring

KW - Hierarchical classification

KW - Writing assessment

UR - http://www.scopus.com/inward/record.url?scp=84911914341&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84911914341&partnerID=8YFLogxK

U2 - 10.1016/j.asw.2014.09.002

DO - 10.1016/j.asw.2014.09.002

M3 - Article

VL - 23

SP - 35

EP - 59

JO - Assessing Writing

JF - Assessing Writing

SN - 1075-2935

ER -