Predicting human judgments of essay quality in both integrated and independent second language writing samples

A comparison study

Liang Guo, Scott A. Crossley, Danielle McNamara

Research output: Contribution to journalArticle

45 Citations (Scopus)

Abstract

This study explores whether linguistic features can predict second language writing proficiency in the Test of English as a Foreign Language (TOEFL iBT) integrated and independent writing tasks and, if so, whether there are differences and similarities in the two sets of predictive linguistic features. Linguistic features related to lexical sophistication, syntactic complexity, cohesion, and basic text information were investigated in relation to the writing scores for both integrated and independent samples. The results of this study show that linguistic features can be used to significantly predict essay scores in the integrated as well as the independent writing. When comparing across the two writing tasks, there are both similarities and differences in the two sets of predictive features. For instance, lexical sophistication was found to be a significant predictor for both tasks while features such as verbs in 3rd person singular form and semantic similarity were only significant predictors for the integrated task. These findings demonstrate that evaluation of the two writing tasks rely on similar and distinct features, and are at least partially assessed using different linguistic criteria. Implications of these findings for the assessment of second language (L2) writing are also discussed.

Original languageEnglish (US)
Pages (from-to)218-238
Number of pages21
JournalAssessing Writing
Volume18
Issue number3
DOIs
StatePublished - Jul 2013

Fingerprint

language
linguistics
Second Language Writing
foreign language
Linguistic Features
semantics
human being
evaluation
Predictors
Sophistication
TOEFL
Language
Person
Verbs
Evaluation
Cohesion
Writing Proficiency
Syntactic Complexity
Semantic Similarity
English as a Foreign Language

Keywords

  • Computational linguistics
  • Independent writing tasks
  • Integrated writing tasks
  • Writing assessment

ASJC Scopus subject areas

  • Language and Linguistics
  • Education
  • Linguistics and Language

Cite this

Predicting human judgments of essay quality in both integrated and independent second language writing samples : A comparison study. / Guo, Liang; Crossley, Scott A.; McNamara, Danielle.

In: Assessing Writing, Vol. 18, No. 3, 07.2013, p. 218-238.

Research output: Contribution to journalArticle

@article{7bd285ca0fa34f0183e2cd1ad60d1222,
title = "Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study",
abstract = "This study explores whether linguistic features can predict second language writing proficiency in the Test of English as a Foreign Language (TOEFL iBT) integrated and independent writing tasks and, if so, whether there are differences and similarities in the two sets of predictive linguistic features. Linguistic features related to lexical sophistication, syntactic complexity, cohesion, and basic text information were investigated in relation to the writing scores for both integrated and independent samples. The results of this study show that linguistic features can be used to significantly predict essay scores in the integrated as well as the independent writing. When comparing across the two writing tasks, there are both similarities and differences in the two sets of predictive features. For instance, lexical sophistication was found to be a significant predictor for both tasks while features such as verbs in 3rd person singular form and semantic similarity were only significant predictors for the integrated task. These findings demonstrate that evaluation of the two writing tasks rely on similar and distinct features, and are at least partially assessed using different linguistic criteria. Implications of these findings for the assessment of second language (L2) writing are also discussed.",
keywords = "Computational linguistics, Independent writing tasks, Integrated writing tasks, Writing assessment",
author = "Liang Guo and Crossley, {Scott A.} and Danielle McNamara",
year = "2013",
month = "7",
doi = "10.1016/j.asw.2013.05.002",
language = "English (US)",
volume = "18",
pages = "218--238",
journal = "Assessing Writing",
issn = "1075-2935",
publisher = "Elsevier Limited",
number = "3",

}

TY - JOUR

T1 - Predicting human judgments of essay quality in both integrated and independent second language writing samples

T2 - A comparison study

AU - Guo, Liang

AU - Crossley, Scott A.

AU - McNamara, Danielle

PY - 2013/7

Y1 - 2013/7

N2 - This study explores whether linguistic features can predict second language writing proficiency in the Test of English as a Foreign Language (TOEFL iBT) integrated and independent writing tasks and, if so, whether there are differences and similarities in the two sets of predictive linguistic features. Linguistic features related to lexical sophistication, syntactic complexity, cohesion, and basic text information were investigated in relation to the writing scores for both integrated and independent samples. The results of this study show that linguistic features can be used to significantly predict essay scores in the integrated as well as the independent writing. When comparing across the two writing tasks, there are both similarities and differences in the two sets of predictive features. For instance, lexical sophistication was found to be a significant predictor for both tasks while features such as verbs in 3rd person singular form and semantic similarity were only significant predictors for the integrated task. These findings demonstrate that evaluation of the two writing tasks rely on similar and distinct features, and are at least partially assessed using different linguistic criteria. Implications of these findings for the assessment of second language (L2) writing are also discussed.

AB - This study explores whether linguistic features can predict second language writing proficiency in the Test of English as a Foreign Language (TOEFL iBT) integrated and independent writing tasks and, if so, whether there are differences and similarities in the two sets of predictive linguistic features. Linguistic features related to lexical sophistication, syntactic complexity, cohesion, and basic text information were investigated in relation to the writing scores for both integrated and independent samples. The results of this study show that linguistic features can be used to significantly predict essay scores in the integrated as well as the independent writing. When comparing across the two writing tasks, there are both similarities and differences in the two sets of predictive features. For instance, lexical sophistication was found to be a significant predictor for both tasks while features such as verbs in 3rd person singular form and semantic similarity were only significant predictors for the integrated task. These findings demonstrate that evaluation of the two writing tasks rely on similar and distinct features, and are at least partially assessed using different linguistic criteria. Implications of these findings for the assessment of second language (L2) writing are also discussed.

KW - Computational linguistics

KW - Independent writing tasks

KW - Integrated writing tasks

KW - Writing assessment

UR - http://www.scopus.com/inward/record.url?scp=84879524017&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84879524017&partnerID=8YFLogxK

U2 - 10.1016/j.asw.2013.05.002

DO - 10.1016/j.asw.2013.05.002

M3 - Article

VL - 18

SP - 218

EP - 238

JO - Assessing Writing

JF - Assessing Writing

SN - 1075-2935

IS - 3

ER -