Automated Writing Evaluation and Feedback: Multiple Metrics of Efficacy

Joshua Wilson, Rod Roscoe

Research output: Contribution to journalArticle

Abstract

The present study extended research on the effectiveness of automated writing evaluation (AWE) systems. Sixth graders were randomly assigned by classroom to an AWE condition that used Project Essay Grade Writing (n = 56) or a word-processing condition that used Google Docs (n = 58). Effectiveness was evaluated using multiple metrics: writing self-efficacy, holistic writing quality, performance on a state English language arts test, and teachers' perceptions of AWE's social validity. Path analyses showed that after controlling for pretest measures, composing condition had no effect on holistic writing quality, but students in the AWE condition had more positive writing self-efficacy and better performance on the state English language arts test. Posttest writing self-efficacy partially mediated the effect of composing condition on state test performance. Teachers reported positive perceptions of AWE's social validity. Results emphasize the importance of using multiple metrics and considering both contextual factors and AWE implementation methods when evaluating AWE effectiveness.

Original languageEnglish (US)
JournalJournal of Educational Computing Research
DOIs
StatePublished - Jan 1 2019

Fingerprint

Word processing
Students
Feedback
evaluation
self-efficacy
English language
art
performance
teacher
search engine

Keywords

  • automated feedback
  • automated writing evaluation
  • interactive learning environments
  • writing
  • writing self-efficacy

ASJC Scopus subject areas

  • Education
  • Computer Science Applications

Cite this

Automated Writing Evaluation and Feedback : Multiple Metrics of Efficacy. / Wilson, Joshua; Roscoe, Rod.

In: Journal of Educational Computing Research, 01.01.2019.

Research output: Contribution to journalArticle

@article{2caf1dd306ba47c9ae0af213084be6fd,
title = "Automated Writing Evaluation and Feedback: Multiple Metrics of Efficacy",
abstract = "The present study extended research on the effectiveness of automated writing evaluation (AWE) systems. Sixth graders were randomly assigned by classroom to an AWE condition that used Project Essay Grade Writing (n = 56) or a word-processing condition that used Google Docs (n = 58). Effectiveness was evaluated using multiple metrics: writing self-efficacy, holistic writing quality, performance on a state English language arts test, and teachers' perceptions of AWE's social validity. Path analyses showed that after controlling for pretest measures, composing condition had no effect on holistic writing quality, but students in the AWE condition had more positive writing self-efficacy and better performance on the state English language arts test. Posttest writing self-efficacy partially mediated the effect of composing condition on state test performance. Teachers reported positive perceptions of AWE's social validity. Results emphasize the importance of using multiple metrics and considering both contextual factors and AWE implementation methods when evaluating AWE effectiveness.",
keywords = "automated feedback, automated writing evaluation, interactive learning environments, writing, writing self-efficacy",
author = "Joshua Wilson and Rod Roscoe",
year = "2019",
month = "1",
day = "1",
doi = "10.1177/0735633119830764",
language = "English (US)",
journal = "Journal of Educational Computing Research",
issn = "0735-6331",
publisher = "Baywood Publishing Co. Inc.",

}

TY - JOUR

T1 - Automated Writing Evaluation and Feedback

T2 - Multiple Metrics of Efficacy

AU - Wilson, Joshua

AU - Roscoe, Rod

PY - 2019/1/1

Y1 - 2019/1/1

N2 - The present study extended research on the effectiveness of automated writing evaluation (AWE) systems. Sixth graders were randomly assigned by classroom to an AWE condition that used Project Essay Grade Writing (n = 56) or a word-processing condition that used Google Docs (n = 58). Effectiveness was evaluated using multiple metrics: writing self-efficacy, holistic writing quality, performance on a state English language arts test, and teachers' perceptions of AWE's social validity. Path analyses showed that after controlling for pretest measures, composing condition had no effect on holistic writing quality, but students in the AWE condition had more positive writing self-efficacy and better performance on the state English language arts test. Posttest writing self-efficacy partially mediated the effect of composing condition on state test performance. Teachers reported positive perceptions of AWE's social validity. Results emphasize the importance of using multiple metrics and considering both contextual factors and AWE implementation methods when evaluating AWE effectiveness.

AB - The present study extended research on the effectiveness of automated writing evaluation (AWE) systems. Sixth graders were randomly assigned by classroom to an AWE condition that used Project Essay Grade Writing (n = 56) or a word-processing condition that used Google Docs (n = 58). Effectiveness was evaluated using multiple metrics: writing self-efficacy, holistic writing quality, performance on a state English language arts test, and teachers' perceptions of AWE's social validity. Path analyses showed that after controlling for pretest measures, composing condition had no effect on holistic writing quality, but students in the AWE condition had more positive writing self-efficacy and better performance on the state English language arts test. Posttest writing self-efficacy partially mediated the effect of composing condition on state test performance. Teachers reported positive perceptions of AWE's social validity. Results emphasize the importance of using multiple metrics and considering both contextual factors and AWE implementation methods when evaluating AWE effectiveness.

KW - automated feedback

KW - automated writing evaluation

KW - interactive learning environments

KW - writing

KW - writing self-efficacy

UR - http://www.scopus.com/inward/record.url?scp=85063569259&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063569259&partnerID=8YFLogxK

U2 - 10.1177/0735633119830764

DO - 10.1177/0735633119830764

M3 - Article

JO - Journal of Educational Computing Research

JF - Journal of Educational Computing Research

SN - 0735-6331

ER -