Automated summarization evaluation (ASE) using natural language processing tools

Scott A. Crossley, Minkyung Kim, Laura Allen, Danielle McNamara

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Summarization is an effective strategy to promote and enhance learning and deep comprehension of texts. However, summarization is seldom implemented by teachers in classrooms because the manual evaluation of students’ summaries requires time and effort. This problem has led to the development of automated models of summarization quality. However, these models often rely on features derived from expert ratings of student summarizations of specific source texts and are therefore not generalizable to summarizations of new texts. Further, many of the models rely of proprietary tools that are not freely or publicly available, rendering replications difficult. In this study, we introduce an automated summarization evaluation (ASE) model that depends strictly on features of the source text or the summary, allowing for a purely text-based model of quality. This model effectively classifies summaries as either low or high quality with an accuracy above 80%. Importantly, the model was developed on a large number of source texts allowing for generalizability across texts. Further, the features used in this study are freely and publicly available affording replication.

Original languageEnglish (US)
Title of host publicationArtificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings
EditorsSeiji Isotani, Peter Hastings, Amy Ogan, Bruce McLaren, Eva Millán, Rose Luckin
PublisherSpringer Verlag
Pages84-95
Number of pages12
ISBN (Print)9783030232030
DOIs
StatePublished - Jan 1 2019
Event20th International Conference on Artificial Intelligence in Education, AIED 2019 - Chicago, United States
Duration: Jun 25 2019Jun 29 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11625 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference20th International Conference on Artificial Intelligence in Education, AIED 2019
CountryUnited States
CityChicago
Period6/25/196/29/19

Keywords

  • Discourse
  • Machine learning
  • Natural language processing
  • Summarization
  • Summary scoring
  • Writing

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Automated summarization evaluation (ASE) using natural language processing tools'. Together they form a unique fingerprint.

  • Cite this

    Crossley, S. A., Kim, M., Allen, L., & McNamara, D. (2019). Automated summarization evaluation (ASE) using natural language processing tools. In S. Isotani, P. Hastings, A. Ogan, B. McLaren, E. Millán, & R. Luckin (Eds.), Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings (pp. 84-95). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11625 LNAI). Springer Verlag. https://doi.org/10.1007/978-3-030-23204-7_8