Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing

Scott A. Crossley, Danielle McNamara

Research output: Contribution to journalArticle

40 Citations (Scopus)

Abstract

This article reviews recent studies in which human judgements of essay quality are assessed using Coh-Metrix, an automated text analysis tool. The goal of these studies is to better understand the relationship between linguistic features of essays and human judgements of writing quality. Coh-Metrix reports on a wide range of linguistic features, affording analyses of writing at various levels of text structure, including surface, text-base, and situation model levels. Recent studies have examined linguistic features of essay quality related to co-reference, connectives, syntactic complexity, lexical diversity, spatiality, temporality, and lexical characteristics. These studies have analysed essays written by both first language and second language writers. The results support the notion that human judgements of essay quality are best predicted by linguistic indices that correlate with measures of language sophistication such as lexical diversity, word frequency, and syntactic complexity. In contrast, human judgements of essay quality are not strongly predicted by linguistic indices related to cohesion. Overall, the studies portray high quality writing as containing more complex language that may not facilitate text comprehension.

Original languageEnglish (US)
Pages (from-to)170-191
Number of pages22
JournalInternational Journal of Continuing Engineering Education and Life-Long Learning
Volume21
Issue number2-3
DOIs
StatePublished - May 2011
Externally publishedYes

Fingerprint

Linguistics
rating
expert
linguistics
language
Syntactics
text analysis
comprehension
writer

Keywords

  • Cohesion
  • Computational linguistics
  • Essay quality
  • Lexical development
  • Syntactic development
  • Writing assessment

ASJC Scopus subject areas

  • Engineering(all)
  • Education

Cite this

@article{95062f342b5743e6a73587466214783e,
title = "Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing",
abstract = "This article reviews recent studies in which human judgements of essay quality are assessed using Coh-Metrix, an automated text analysis tool. The goal of these studies is to better understand the relationship between linguistic features of essays and human judgements of writing quality. Coh-Metrix reports on a wide range of linguistic features, affording analyses of writing at various levels of text structure, including surface, text-base, and situation model levels. Recent studies have examined linguistic features of essay quality related to co-reference, connectives, syntactic complexity, lexical diversity, spatiality, temporality, and lexical characteristics. These studies have analysed essays written by both first language and second language writers. The results support the notion that human judgements of essay quality are best predicted by linguistic indices that correlate with measures of language sophistication such as lexical diversity, word frequency, and syntactic complexity. In contrast, human judgements of essay quality are not strongly predicted by linguistic indices related to cohesion. Overall, the studies portray high quality writing as containing more complex language that may not facilitate text comprehension.",
keywords = "Cohesion, Computational linguistics, Essay quality, Lexical development, Syntactic development, Writing assessment",
author = "Crossley, {Scott A.} and Danielle McNamara",
year = "2011",
month = "5",
doi = "10.1504/IJCEELL.2011.040197",
language = "English (US)",
volume = "21",
pages = "170--191",
journal = "International Journal of Continuing Engineering Education and Life-Long Learning",
issn = "1560-4624",
publisher = "Inderscience Enterprises Ltd",
number = "2-3",

}

TY - JOUR

T1 - Understanding expert ratings of essay quality

T2 - Coh-Metrix analyses of first and second language writing

AU - Crossley, Scott A.

AU - McNamara, Danielle

PY - 2011/5

Y1 - 2011/5

N2 - This article reviews recent studies in which human judgements of essay quality are assessed using Coh-Metrix, an automated text analysis tool. The goal of these studies is to better understand the relationship between linguistic features of essays and human judgements of writing quality. Coh-Metrix reports on a wide range of linguistic features, affording analyses of writing at various levels of text structure, including surface, text-base, and situation model levels. Recent studies have examined linguistic features of essay quality related to co-reference, connectives, syntactic complexity, lexical diversity, spatiality, temporality, and lexical characteristics. These studies have analysed essays written by both first language and second language writers. The results support the notion that human judgements of essay quality are best predicted by linguistic indices that correlate with measures of language sophistication such as lexical diversity, word frequency, and syntactic complexity. In contrast, human judgements of essay quality are not strongly predicted by linguistic indices related to cohesion. Overall, the studies portray high quality writing as containing more complex language that may not facilitate text comprehension.

AB - This article reviews recent studies in which human judgements of essay quality are assessed using Coh-Metrix, an automated text analysis tool. The goal of these studies is to better understand the relationship between linguistic features of essays and human judgements of writing quality. Coh-Metrix reports on a wide range of linguistic features, affording analyses of writing at various levels of text structure, including surface, text-base, and situation model levels. Recent studies have examined linguistic features of essay quality related to co-reference, connectives, syntactic complexity, lexical diversity, spatiality, temporality, and lexical characteristics. These studies have analysed essays written by both first language and second language writers. The results support the notion that human judgements of essay quality are best predicted by linguistic indices that correlate with measures of language sophistication such as lexical diversity, word frequency, and syntactic complexity. In contrast, human judgements of essay quality are not strongly predicted by linguistic indices related to cohesion. Overall, the studies portray high quality writing as containing more complex language that may not facilitate text comprehension.

KW - Cohesion

KW - Computational linguistics

KW - Essay quality

KW - Lexical development

KW - Syntactic development

KW - Writing assessment

UR - http://www.scopus.com/inward/record.url?scp=79957818785&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79957818785&partnerID=8YFLogxK

U2 - 10.1504/IJCEELL.2011.040197

DO - 10.1504/IJCEELL.2011.040197

M3 - Article

AN - SCOPUS:79957818785

VL - 21

SP - 170

EP - 191

JO - International Journal of Continuing Engineering Education and Life-Long Learning

JF - International Journal of Continuing Engineering Education and Life-Long Learning

SN - 1560-4624

IS - 2-3

ER -