TY - JOUR
T1 - Automated Writing Evaluation and Feedback
T2 - Multiple Metrics of Efficacy
AU - Wilson, Joshua
AU - Roscoe, Rod
PY - 2019/1/1
Y1 - 2019/1/1
N2 - The present study extended research on the effectiveness of automated writing evaluation (AWE) systems. Sixth graders were randomly assigned by classroom to an AWE condition that used Project Essay Grade Writing (n = 56) or a word-processing condition that used Google Docs (n = 58). Effectiveness was evaluated using multiple metrics: writing self-efficacy, holistic writing quality, performance on a state English language arts test, and teachers' perceptions of AWE's social validity. Path analyses showed that after controlling for pretest measures, composing condition had no effect on holistic writing quality, but students in the AWE condition had more positive writing self-efficacy and better performance on the state English language arts test. Posttest writing self-efficacy partially mediated the effect of composing condition on state test performance. Teachers reported positive perceptions of AWE's social validity. Results emphasize the importance of using multiple metrics and considering both contextual factors and AWE implementation methods when evaluating AWE effectiveness.
AB - The present study extended research on the effectiveness of automated writing evaluation (AWE) systems. Sixth graders were randomly assigned by classroom to an AWE condition that used Project Essay Grade Writing (n = 56) or a word-processing condition that used Google Docs (n = 58). Effectiveness was evaluated using multiple metrics: writing self-efficacy, holistic writing quality, performance on a state English language arts test, and teachers' perceptions of AWE's social validity. Path analyses showed that after controlling for pretest measures, composing condition had no effect on holistic writing quality, but students in the AWE condition had more positive writing self-efficacy and better performance on the state English language arts test. Posttest writing self-efficacy partially mediated the effect of composing condition on state test performance. Teachers reported positive perceptions of AWE's social validity. Results emphasize the importance of using multiple metrics and considering both contextual factors and AWE implementation methods when evaluating AWE effectiveness.
KW - automated feedback
KW - automated writing evaluation
KW - interactive learning environments
KW - writing
KW - writing self-efficacy
UR - http://www.scopus.com/inward/record.url?scp=85063569259&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85063569259&partnerID=8YFLogxK
U2 - 10.1177/0735633119830764
DO - 10.1177/0735633119830764
M3 - Article
AN - SCOPUS:85063569259
SN - 0735-6331
JO - Journal of Educational Computing Research
JF - Journal of Educational Computing Research
ER -