TY - GEN
T1 - Scoring summaries using recurrent neural networks
AU - Ruseti, Stefan
AU - Dascalu, Mihai
AU - Johnson, Amy
AU - McNamara, Danielle
AU - Balyan, Renu
AU - McCarthy, Kathryn S.
AU - Trausan-Matu, Stefan
N1 - Funding Information:
Acknowledgment. This research was partially supported by the README project “Interactive and Innovative application for evaluating the readability of texts in Romanian Language and for improving users’ writing styles”, contract no. 114/15.09.2017, MySMIS 2014 code 119286, the 644187 EC H2020 RAGE project, the FP7 2008-212578 LTfLL project, the Department of Education, Institute of Education Sciences - Grant R305A130124, as well as the Department of Defense, Office of Naval Research - Grants N00014140343 and N000141712300.
Funding Information:
This research was partially supported by the README project “Interactive and Innovative application for evaluating the readability of texts in Romanian Language and for improving users’ writing styles”, contract no. 114/15.09.2017, MySMIS 2014 code 119286, the 644187 EC H2020 RAGE project, the FP7 2008-212578 LTfLL project, the Department of Education, Institute of Education Sciences-Grant R305A130124, as well as the Department of Defense, Office of Naval Research-Grants N00014140343 and N000141712300.
Publisher Copyright:
© Springer International Publishing AG, part of Springer Nature 2018.
PY - 2018
Y1 - 2018
N2 - Summarization enhances comprehension and is considered an effective strategy to promote and enhance learning and deep understanding of texts. However, summarization is seldom implemented by teachers in classrooms because the manual evaluation requires a lot of effort and time. Although the need for automated support is stringent, there are only a few shallow systems available, most of which rely on basic word/n-gram overlaps. In this paper, we introduce a hybrid model that uses state-of-the-art recurrent neural networks and textual complexity indices to score summaries. Our best model achieves over 55% accuracy for a 3-way classification that measures the degree to which the main ideas from the original text are covered by the summary. Our experiments show that the writing style, represented by the textual complexity indices, together with the semantic content grasped within the summary are the best predictors, when combined. To the best of our knowledge, this is the first work of its kind that uses RNNs for scoring and evaluating summaries.
AB - Summarization enhances comprehension and is considered an effective strategy to promote and enhance learning and deep understanding of texts. However, summarization is seldom implemented by teachers in classrooms because the manual evaluation requires a lot of effort and time. Although the need for automated support is stringent, there are only a few shallow systems available, most of which rely on basic word/n-gram overlaps. In this paper, we introduce a hybrid model that uses state-of-the-art recurrent neural networks and textual complexity indices to score summaries. Our best model achieves over 55% accuracy for a 3-way classification that measures the degree to which the main ideas from the original text are covered by the summary. Our experiments show that the writing style, represented by the textual complexity indices, together with the semantic content grasped within the summary are the best predictors, when combined. To the best of our knowledge, this is the first work of its kind that uses RNNs for scoring and evaluating summaries.
KW - Automated summary evaluation
KW - Recurrent neural network
KW - Semantic models
KW - Word embeddings
UR - http://www.scopus.com/inward/record.url?scp=85048346442&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85048346442&partnerID=8YFLogxK
U2 - 10.1007/978-3-319-91464-0_19
DO - 10.1007/978-3-319-91464-0_19
M3 - Conference contribution
AN - SCOPUS:85048346442
SN - 9783319914633
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 191
EP - 201
BT - Intelligent Tutoring Systems - 14th International Conference, ITS 2018, Proceedings
A2 - Vassileva, Julita
A2 - Nkambou, Roger
A2 - Azevedo, Roger
PB - Springer Verlag
T2 - 14th International Conference on Intelligent Tutoring Systems, ITS 2018
Y2 - 11 June 2018 through 15 June 2018
ER -