Combining machine learning and natural language processing to assess literary text comprehension

Renu Balyan, Kathryn S. McCarthy, Danielle S. McNamara

Research output: Contribution to conferencePaper

4 Scopus citations

Abstract

This study examined how machine learning and natural language processing (NLP) techniques can be leveraged to assess the interpretive behavior that is required for successful literary text comprehension. We compared the accuracy of seven different machine learning classification algorithms in predicting human ratings of student essays about literary works. Three types of NLP feature sets: unigrams (single content words), elaborative (new) n-grams, and linguistic features were used to classify idea units (paraphrase, text-based inference, interpretive inference). The most accurate classifications emerged using all three NLP features sets in combination, with accuracy ranging from 0.61 to 0.94 (F=0.18 to 0.81). Random Forests, which employs multiple decision trees and a bagging approach, was the most accurate classifier for these data. In contrast, the single classifier, Trees, which tends to “overfit” the data during training, was the least accurate. Ensemble classifiers were generally more accurate than single classifiers. However, Support Vector Machines accuracy was comparable to that of the ensemble classifiers. This is likely due to Support Vector Machines’ unique ability to support high dimension feature spaces. The findings suggest that combining the power of NLP and machine learning is an effective means of automating literary text comprehension assessment.

Original languageEnglish (US)
Pages244-249
Number of pages6
StatePublished - Jan 1 2017
Event10th International Conference on Educational Data Mining, EDM 2017 - Wuhan, China
Duration: Jun 25 2017Jun 28 2017

Conference

Conference10th International Conference on Educational Data Mining, EDM 2017
CountryChina
CityWuhan
Period6/25/176/28/17

Keywords

  • Classification
  • Interpretation
  • Natural language processing
  • Supervised machine learning

ASJC Scopus subject areas

  • Computer Science Applications
  • Information Systems

Fingerprint Dive into the research topics of 'Combining machine learning and natural language processing to assess literary text comprehension'. Together they form a unique fingerprint.

  • Cite this

    Balyan, R., McCarthy, K. S., & McNamara, D. S. (2017). Combining machine learning and natural language processing to assess literary text comprehension. 244-249. Paper presented at 10th International Conference on Educational Data Mining, EDM 2017, Wuhan, China.