Automated Summary Scoring with ReaderBench

Robert Mihai Botarleanu, Mihai Dascalu, Laura K. Allen, Scott Andrew Crossley, Danielle S. McNamara

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Text summarization is an effective reading comprehension strategy. However, summary evaluation is complex and must account for various factors including the summary and the reference text. This study examines a corpus of approximately 3,000 summaries based on 87 reference texts, with each summary being manually scored on a 4-point Likert scale. Machine learning models leveraging Natural Language Processing (NLP) techniques were trained to predict the extent to which summaries capture the main idea of the target text. The NLP models combined both domain and language independent textual complexity indices from the ReaderBench framework, as well as state-of-the-art language models and deep learning architectures to provide semantic contextualization. The models achieve low errors – normalized MAE ranging from 0.13–0.17 with corresponding R2 values of up to 0.46. Our approach consistently outperforms baselines that use TF-IDF vectors and linear models, as well as Transfomer-based regression using BERT. These results indicate that NLP algorithms that combine linguistic and semantic indices are accurate and robust, while ensuring generalizability to a wide array of topics.

Original languageEnglish (US)
Title of host publicationIntelligent Tutoring Systems - 17th International Conference, ITS 2021, Proceedings
EditorsAlexandra I. Cristea, Christos Troussas
PublisherSpringer Science and Business Media Deutschland GmbH
Pages321-332
Number of pages12
ISBN (Print)9783030804206
DOIs
StatePublished - 2021
Externally publishedYes
Event17th International Conference on Intelligent Tutoring Systems, ITS 2021 - Virtual, Online
Duration: Jun 7 2021Jun 11 2021

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12677 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference17th International Conference on Intelligent Tutoring Systems, ITS 2021
CityVirtual, Online
Period6/7/216/11/21

Keywords

  • Automated scoring
  • Natural language processing
  • Text summarization

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'Automated Summary Scoring with ReaderBench'. Together they form a unique fingerprint.

Cite this