Automated summarization evaluation (ASE) using natural language processing tools

Scott A. Crossley, Minkyung Kim, Laura Allen, Danielle McNamara

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Summarization is an effective strategy to promote and enhance learning and deep comprehension of texts. However, summarization is seldom implemented by teachers in classrooms because the manual evaluation of students’ summaries requires time and effort. This problem has led to the development of automated models of summarization quality. However, these models often rely on features derived from expert ratings of student summarizations of specific source texts and are therefore not generalizable to summarizations of new texts. Further, many of the models rely of proprietary tools that are not freely or publicly available, rendering replications difficult. In this study, we introduce an automated summarization evaluation (ASE) model that depends strictly on features of the source text or the summary, allowing for a purely text-based model of quality. This model effectively classifies summaries as either low or high quality with an accuracy above 80%. Importantly, the model was developed on a large number of source texts allowing for generalizability across texts. Further, the features used in this study are freely and publicly available affording replication.

Original languageEnglish (US)
Title of host publicationArtificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings
EditorsSeiji Isotani, Peter Hastings, Amy Ogan, Bruce McLaren, Eva Millán, Rose Luckin
PublisherSpringer Verlag
Pages84-95
Number of pages12
ISBN (Print)9783030232030
DOIs
StatePublished - Jan 1 2019
Event20th International Conference on Artificial Intelligence in Education, AIED 2019 - Chicago, United States
Duration: Jun 25 2019Jun 29 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11625 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference20th International Conference on Artificial Intelligence in Education, AIED 2019
CountryUnited States
CityChicago
Period6/25/196/29/19

Fingerprint

Summarization
Natural Language
Evaluation
Processing
Replication
Model
Students
Evaluation Model
Text
Rendering
Strictly
Classify

Keywords

  • Discourse
  • Machine learning
  • Natural language processing
  • Summarization
  • Summary scoring
  • Writing

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Crossley, S. A., Kim, M., Allen, L., & McNamara, D. (2019). Automated summarization evaluation (ASE) using natural language processing tools. In S. Isotani, P. Hastings, A. Ogan, B. McLaren, E. Millán, & R. Luckin (Eds.), Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings (pp. 84-95). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11625 LNAI). Springer Verlag. https://doi.org/10.1007/978-3-030-23204-7_8

Automated summarization evaluation (ASE) using natural language processing tools. / Crossley, Scott A.; Kim, Minkyung; Allen, Laura; McNamara, Danielle.

Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings. ed. / Seiji Isotani; Peter Hastings; Amy Ogan; Bruce McLaren; Eva Millán; Rose Luckin. Springer Verlag, 2019. p. 84-95 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11625 LNAI).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Crossley, SA, Kim, M, Allen, L & McNamara, D 2019, Automated summarization evaluation (ASE) using natural language processing tools. in S Isotani, P Hastings, A Ogan, B McLaren, E Millán & R Luckin (eds), Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11625 LNAI, Springer Verlag, pp. 84-95, 20th International Conference on Artificial Intelligence in Education, AIED 2019, Chicago, United States, 6/25/19. https://doi.org/10.1007/978-3-030-23204-7_8
Crossley SA, Kim M, Allen L, McNamara D. Automated summarization evaluation (ASE) using natural language processing tools. In Isotani S, Hastings P, Ogan A, McLaren B, Millán E, Luckin R, editors, Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings. Springer Verlag. 2019. p. 84-95. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-23204-7_8
Crossley, Scott A. ; Kim, Minkyung ; Allen, Laura ; McNamara, Danielle. / Automated summarization evaluation (ASE) using natural language processing tools. Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings. editor / Seiji Isotani ; Peter Hastings ; Amy Ogan ; Bruce McLaren ; Eva Millán ; Rose Luckin. Springer Verlag, 2019. pp. 84-95 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{481cbe1c856f4f75aac325f416c0d7a5,
title = "Automated summarization evaluation (ASE) using natural language processing tools",
abstract = "Summarization is an effective strategy to promote and enhance learning and deep comprehension of texts. However, summarization is seldom implemented by teachers in classrooms because the manual evaluation of students’ summaries requires time and effort. This problem has led to the development of automated models of summarization quality. However, these models often rely on features derived from expert ratings of student summarizations of specific source texts and are therefore not generalizable to summarizations of new texts. Further, many of the models rely of proprietary tools that are not freely or publicly available, rendering replications difficult. In this study, we introduce an automated summarization evaluation (ASE) model that depends strictly on features of the source text or the summary, allowing for a purely text-based model of quality. This model effectively classifies summaries as either low or high quality with an accuracy above 80{\%}. Importantly, the model was developed on a large number of source texts allowing for generalizability across texts. Further, the features used in this study are freely and publicly available affording replication.",
keywords = "Discourse, Machine learning, Natural language processing, Summarization, Summary scoring, Writing",
author = "Crossley, {Scott A.} and Minkyung Kim and Laura Allen and Danielle McNamara",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/978-3-030-23204-7_8",
language = "English (US)",
isbn = "9783030232030",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "84--95",
editor = "Seiji Isotani and Peter Hastings and Amy Ogan and Bruce McLaren and Eva Mill{\'a}n and Rose Luckin",
booktitle = "Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings",

}

TY - GEN

T1 - Automated summarization evaluation (ASE) using natural language processing tools

AU - Crossley, Scott A.

AU - Kim, Minkyung

AU - Allen, Laura

AU - McNamara, Danielle

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Summarization is an effective strategy to promote and enhance learning and deep comprehension of texts. However, summarization is seldom implemented by teachers in classrooms because the manual evaluation of students’ summaries requires time and effort. This problem has led to the development of automated models of summarization quality. However, these models often rely on features derived from expert ratings of student summarizations of specific source texts and are therefore not generalizable to summarizations of new texts. Further, many of the models rely of proprietary tools that are not freely or publicly available, rendering replications difficult. In this study, we introduce an automated summarization evaluation (ASE) model that depends strictly on features of the source text or the summary, allowing for a purely text-based model of quality. This model effectively classifies summaries as either low or high quality with an accuracy above 80%. Importantly, the model was developed on a large number of source texts allowing for generalizability across texts. Further, the features used in this study are freely and publicly available affording replication.

AB - Summarization is an effective strategy to promote and enhance learning and deep comprehension of texts. However, summarization is seldom implemented by teachers in classrooms because the manual evaluation of students’ summaries requires time and effort. This problem has led to the development of automated models of summarization quality. However, these models often rely on features derived from expert ratings of student summarizations of specific source texts and are therefore not generalizable to summarizations of new texts. Further, many of the models rely of proprietary tools that are not freely or publicly available, rendering replications difficult. In this study, we introduce an automated summarization evaluation (ASE) model that depends strictly on features of the source text or the summary, allowing for a purely text-based model of quality. This model effectively classifies summaries as either low or high quality with an accuracy above 80%. Importantly, the model was developed on a large number of source texts allowing for generalizability across texts. Further, the features used in this study are freely and publicly available affording replication.

KW - Discourse

KW - Machine learning

KW - Natural language processing

KW - Summarization

KW - Summary scoring

KW - Writing

UR - http://www.scopus.com/inward/record.url?scp=85068309248&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85068309248&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-23204-7_8

DO - 10.1007/978-3-030-23204-7_8

M3 - Conference contribution

SN - 9783030232030

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 84

EP - 95

BT - Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings

A2 - Isotani, Seiji

A2 - Hastings, Peter

A2 - Ogan, Amy

A2 - McLaren, Bruce

A2 - Millán, Eva

A2 - Luckin, Rose

PB - Springer Verlag

ER -