The Development of the Writing Assessment Tool: An On-line Platform for the Automated Assessment of Writing

Project: Research project

Description

Summary

The Development of the Writing Assessment Tool (WAT):
An On-line Platform for the Automated Assessment of Writing

Purpose: The aim of this Goal 2, Development and Innovation project is to develop the Writing Assessment Tool (WAT): an on-line platform that provides students, teachers, and researchers access to automated writing analytics. WAT will comprise three access points, each tailored to the needs of these three end-users. From a single entry point: Students will receive automated summative and formative feedback on three types of essays: persuasive (independent) essays, summaries, and source-based (integrative) essays. Teachers will have access to a teacher interface allowing them to administer essay assignments, which they can choose to be automatically scored or grade themselves using scaffolded rubrics. Researchers will have access to a web-based tool, a downloadable tool, and editable software, which will allow them to conduct computational analyses of writing. Our overall aim is to develop a tool that will have a broad impact on current practices in writing research and instruction across multiple dimensions.
Setting: WAT development will be carried out across three universities: Arizona State University, Mississippi State University, and Georgia State University. Focus groups, usability studies, and design-based classroom studies will be conducted at ASU and high schools in the Georgia and Mississippi participating school district areas.
Sample/Population: Participants in focus groups and usability studies will include high school students and teachers, as well as writing researchers. Teachers in the Phoenix, Atlanta, and Starkville areas will serve as partners throughout the duration of the project, collaborating in the design and implementation of WAT in their classrooms. Participants in a Year 4 feasibility study will include approximately 1,000 (socioeconomically and ethnically diverse) high school students across a minimum of 25 classrooms.
Research Design and Methods: In Year 4, we will conduct a classroom feasibility study to explore the implementation of WAT in authentic classrooms. The study will be a pretest-posttest quasi-experimental design and will not include a no-treatment comparison group. Because learning is expected to be strongest over multiple system interactions, the study will be conducted over a 14-week period. We will collect multiple sources of data (e.g., surveys, log data) to assess the impact of WAT within the real-world constraints and needs of the classroom.
Key Measures: Our theory of change includes as its endpoint the ultimate objective to enhance students writing ability to compose high quality writing across multiple genres. Therefore, a feasibility study will investigate the degree to which the quality of students persuasive essays, summaries, and source-based essays improve as function of interacting with WAT. Additionally, we expect WAT to improve metacognitive knowledge of writing across multiple dimensions, which will be assessed via surveys distributed before and after users interact with WAT during usability studies.
Data Analytic Strategy: We will rely on inferential and data mining analysis techniques to develop a robust understanding of the impact of WAT on students, teachers, and researchers. Inferential statistical analyses will serve as our primary means of documenting end-users WAT usage and assessing learning gains. These analyses will primarily focus on the pretest-posttest data collected in the Year 4 feasibility study. Data mining analyses will predominantly rely on log data (e.g., keystroke logs) to identify fine-grained behavioral patterns of WAT use. The purpose of these inferential and data mining analyses is to find associations between components of WAT and student outcomes, such that this information can be used to support the transition of the system into classrooms and increase the likelihood of a successful efficacy evaluation.
StatusActive
Effective start/end date7/1/186/30/22

Funding

  • US Department of Education (DOEd): $1,399,327.00

Fingerprint

classroom
student
teacher
school
student teacher
study group
learning
research planning
research method
genre
Group
district
instruction
innovation
university
ability
interaction
evaluation
knowledge