Using automated indices of cohesion to evaluate an intelligent tutoring system and an automated writing evaluation system

Scott A. Crossley, Laura K. Varner, Rod Roscoe, Danielle McNamara

Research output: Chapter in Book/Report/Conference proceedingConference contribution

14 Citations (Scopus)

Abstract

We present an evaluation of the Writing Pal (W-Pal) intelligent tutoring system (ITS) and the W-Pal automated writing evaluation (AWE) system through the use ofcomputational indices related to text cohesion. Sixty-four students participated in this study. Each student was assigned to either the W-Pal ITS condition or the W-Pal AWE condition. The W-Pal ITS includes strategy instruction, game-based practice, and essay-based practice with automated feedback. In the ITS condition, students received strategy training and wrote and revised one essay in each of the 8 training sessions. In the AWE condition, students only interacted with the essay writing and feedback tools. These students wrote and revised two essays in each of the 8 sessions. Indices of local and global cohesion reported by the computational tools Coh-Metrix and the Writing Assessment Tool (WAT) were used to investigate pretest and posttest writing gains. For both the ITS and the AWE systems, training led to the increased use of global cohesion features in essay writing. This study demonstrates that automated indices of text cohesion can be used to evaluate the effects of ITSs and AWE systems and further demonstrates how text cohesion develops as a result of instruction, writing, and automated feedback.

Original languageEnglish (US)
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Pages269-278
Number of pages10
Volume7926 LNAI
DOIs
StatePublished - 2013
Event16th International Conference on Artificial Intelligence in Education, AIED 2013 - Memphis, TN, United States
Duration: Jul 9 2013Jul 13 2013

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume7926 LNAI
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other16th International Conference on Artificial Intelligence in Education, AIED 2013
CountryUnited States
CityMemphis, TN
Period7/9/137/13/13

Fingerprint

Intelligent Tutoring Systems
Cohesion
Intelligent systems
Students
Evaluate
Evaluation
Feedback
Pre-test
Demonstrate
Game
Text
Training

Keywords

  • Cohesion
  • Computational Linguistics
  • Corpus Linguistics
  • Intelligent Tutoring Systems
  • Natural Language Processing
  • Writing Pedagogy

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Crossley, S. A., Varner, L. K., Roscoe, R., & McNamara, D. (2013). Using automated indices of cohesion to evaluate an intelligent tutoring system and an automated writing evaluation system. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 7926 LNAI, pp. 269-278). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7926 LNAI). https://doi.org/10.1007/978-3-642-39112-5-28

Using automated indices of cohesion to evaluate an intelligent tutoring system and an automated writing evaluation system. / Crossley, Scott A.; Varner, Laura K.; Roscoe, Rod; McNamara, Danielle.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 7926 LNAI 2013. p. 269-278 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7926 LNAI).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Crossley, SA, Varner, LK, Roscoe, R & McNamara, D 2013, Using automated indices of cohesion to evaluate an intelligent tutoring system and an automated writing evaluation system. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 7926 LNAI, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 7926 LNAI, pp. 269-278, 16th International Conference on Artificial Intelligence in Education, AIED 2013, Memphis, TN, United States, 7/9/13. https://doi.org/10.1007/978-3-642-39112-5-28
Crossley SA, Varner LK, Roscoe R, McNamara D. Using automated indices of cohesion to evaluate an intelligent tutoring system and an automated writing evaluation system. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 7926 LNAI. 2013. p. 269-278. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-642-39112-5-28
Crossley, Scott A. ; Varner, Laura K. ; Roscoe, Rod ; McNamara, Danielle. / Using automated indices of cohesion to evaluate an intelligent tutoring system and an automated writing evaluation system. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 7926 LNAI 2013. pp. 269-278 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{15a08f7459e8418c919feae4b8adb780,
title = "Using automated indices of cohesion to evaluate an intelligent tutoring system and an automated writing evaluation system",
abstract = "We present an evaluation of the Writing Pal (W-Pal) intelligent tutoring system (ITS) and the W-Pal automated writing evaluation (AWE) system through the use ofcomputational indices related to text cohesion. Sixty-four students participated in this study. Each student was assigned to either the W-Pal ITS condition or the W-Pal AWE condition. The W-Pal ITS includes strategy instruction, game-based practice, and essay-based practice with automated feedback. In the ITS condition, students received strategy training and wrote and revised one essay in each of the 8 training sessions. In the AWE condition, students only interacted with the essay writing and feedback tools. These students wrote and revised two essays in each of the 8 sessions. Indices of local and global cohesion reported by the computational tools Coh-Metrix and the Writing Assessment Tool (WAT) were used to investigate pretest and posttest writing gains. For both the ITS and the AWE systems, training led to the increased use of global cohesion features in essay writing. This study demonstrates that automated indices of text cohesion can be used to evaluate the effects of ITSs and AWE systems and further demonstrates how text cohesion develops as a result of instruction, writing, and automated feedback.",
keywords = "Cohesion, Computational Linguistics, Corpus Linguistics, Intelligent Tutoring Systems, Natural Language Processing, Writing Pedagogy",
author = "Crossley, {Scott A.} and Varner, {Laura K.} and Rod Roscoe and Danielle McNamara",
year = "2013",
doi = "10.1007/978-3-642-39112-5-28",
language = "English (US)",
isbn = "9783642391118",
volume = "7926 LNAI",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
pages = "269--278",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

TY - GEN

T1 - Using automated indices of cohesion to evaluate an intelligent tutoring system and an automated writing evaluation system

AU - Crossley, Scott A.

AU - Varner, Laura K.

AU - Roscoe, Rod

AU - McNamara, Danielle

PY - 2013

Y1 - 2013

N2 - We present an evaluation of the Writing Pal (W-Pal) intelligent tutoring system (ITS) and the W-Pal automated writing evaluation (AWE) system through the use ofcomputational indices related to text cohesion. Sixty-four students participated in this study. Each student was assigned to either the W-Pal ITS condition or the W-Pal AWE condition. The W-Pal ITS includes strategy instruction, game-based practice, and essay-based practice with automated feedback. In the ITS condition, students received strategy training and wrote and revised one essay in each of the 8 training sessions. In the AWE condition, students only interacted with the essay writing and feedback tools. These students wrote and revised two essays in each of the 8 sessions. Indices of local and global cohesion reported by the computational tools Coh-Metrix and the Writing Assessment Tool (WAT) were used to investigate pretest and posttest writing gains. For both the ITS and the AWE systems, training led to the increased use of global cohesion features in essay writing. This study demonstrates that automated indices of text cohesion can be used to evaluate the effects of ITSs and AWE systems and further demonstrates how text cohesion develops as a result of instruction, writing, and automated feedback.

AB - We present an evaluation of the Writing Pal (W-Pal) intelligent tutoring system (ITS) and the W-Pal automated writing evaluation (AWE) system through the use ofcomputational indices related to text cohesion. Sixty-four students participated in this study. Each student was assigned to either the W-Pal ITS condition or the W-Pal AWE condition. The W-Pal ITS includes strategy instruction, game-based practice, and essay-based practice with automated feedback. In the ITS condition, students received strategy training and wrote and revised one essay in each of the 8 training sessions. In the AWE condition, students only interacted with the essay writing and feedback tools. These students wrote and revised two essays in each of the 8 sessions. Indices of local and global cohesion reported by the computational tools Coh-Metrix and the Writing Assessment Tool (WAT) were used to investigate pretest and posttest writing gains. For both the ITS and the AWE systems, training led to the increased use of global cohesion features in essay writing. This study demonstrates that automated indices of text cohesion can be used to evaluate the effects of ITSs and AWE systems and further demonstrates how text cohesion develops as a result of instruction, writing, and automated feedback.

KW - Cohesion

KW - Computational Linguistics

KW - Corpus Linguistics

KW - Intelligent Tutoring Systems

KW - Natural Language Processing

KW - Writing Pedagogy

UR - http://www.scopus.com/inward/record.url?scp=84880012839&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84880012839&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-39112-5-28

DO - 10.1007/978-3-642-39112-5-28

M3 - Conference contribution

AN - SCOPUS:84880012839

SN - 9783642391118

VL - 7926 LNAI

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 269

EP - 278

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

ER -