Multitask Summary Scoring with Longformers

Robert Mihai Botarleanu, Mihai Dascalu, Laura K. Allen, Scott Andrew Crossley, Danielle S. McNamara

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Automated scoring of student language is a complex task that requires systems to emulate complex and multi-faceted human evaluation criteria. Summary scoring brings an additional layer of complexity to automated scoring because it involves two texts of differing lengths that must be compared. In this study, we present our approach to automate summary scoring by evaluating a corpus of approximately 5,000 summaries based on 103 source texts, each summary being scored on a 4-point Likert scale for seven different evaluation criteria. We train and evaluate a series of Machine Learning models that use a combination of independent textual complexity indices from the ReaderBench framework and Deep Learning models based on the Transformer architecture in a multitask setup to predict concurrently all criteria. Our models achieve significantly lower errors than previous work using a similar dataset, with MAE ranging from 0.10–0.16 and corresponding R2 values of up to 0.64. Our findings indicate that Longformer-based [1] models are adequate for contextualizing longer text sequences and effectively scoring summaries according to a variety of human-defined evaluation criteria using a single Neural Network.

Original languageEnglish (US)
Title of host publicationArtificial Intelligence in Education - 23rd International Conference, AIED 2022, Proceedings
EditorsMaria Mercedes Rodrigo, Noburu Matsuda, Alexandra I. Cristea, Vania Dimitrova
PublisherSpringer Science and Business Media Deutschland GmbH
Pages756-761
Number of pages6
ISBN (Print)9783031116438
DOIs
StatePublished - 2022
Event23rd International Conference on Artificial Intelligence in Education, AIED 2022 - Durham, United Kingdom
Duration: Jul 27 2022Jul 31 2022

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13355 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference23rd International Conference on Artificial Intelligence in Education, AIED 2022
Country/TerritoryUnited Kingdom
CityDurham
Period7/27/227/31/22

Keywords

  • Automated summary scoring
  • Multitask learning
  • Natural language processing
  • Text summarization

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'Multitask Summary Scoring with Longformers'. Together they form a unique fingerprint.

Cite this