Combining machine learning and natural language processing to assess literary text comprehension

Renu Balyan, Kathryn S. McCarthy, Danielle S. McNamara

Research output: Contribution to conferencePaper

3 Citations (Scopus)

Abstract

This study examined how machine learning and natural language processing (NLP) techniques can be leveraged to assess the interpretive behavior that is required for successful literary text comprehension. We compared the accuracy of seven different machine learning classification algorithms in predicting human ratings of student essays about literary works. Three types of NLP feature sets: unigrams (single content words), elaborative (new) n-grams, and linguistic features were used to classify idea units (paraphrase, text-based inference, interpretive inference). The most accurate classifications emerged using all three NLP features sets in combination, with accuracy ranging from 0.61 to 0.94 (F=0.18 to 0.81). Random Forests, which employs multiple decision trees and a bagging approach, was the most accurate classifier for these data. In contrast, the single classifier, Trees, which tends to “overfit” the data during training, was the least accurate. Ensemble classifiers were generally more accurate than single classifiers. However, Support Vector Machines accuracy was comparable to that of the ensemble classifiers. This is likely due to Support Vector Machines’ unique ability to support high dimension feature spaces. The findings suggest that combining the power of NLP and machine learning is an effective means of automating literary text comprehension assessment.

Original languageEnglish (US)
Pages244-249
Number of pages6
StatePublished - Jan 1 2017
Externally publishedYes
Event10th International Conference on Educational Data Mining, EDM 2017 - Wuhan, China
Duration: Jun 25 2017Jun 28 2017

Conference

Conference10th International Conference on Educational Data Mining, EDM 2017
CountryChina
CityWuhan
Period6/25/176/28/17

Fingerprint

Learning systems
Classifiers
Processing
Support vector machines
Decision trees
Linguistics
Students

Keywords

  • Classification
  • Interpretation
  • Natural language processing
  • Supervised machine learning

ASJC Scopus subject areas

  • Computer Science Applications
  • Information Systems

Cite this

Balyan, R., McCarthy, K. S., & McNamara, D. S. (2017). Combining machine learning and natural language processing to assess literary text comprehension. 244-249. Paper presented at 10th International Conference on Educational Data Mining, EDM 2017, Wuhan, China.

Combining machine learning and natural language processing to assess literary text comprehension. / Balyan, Renu; McCarthy, Kathryn S.; McNamara, Danielle S.

2017. 244-249 Paper presented at 10th International Conference on Educational Data Mining, EDM 2017, Wuhan, China.

Research output: Contribution to conferencePaper

Balyan, R, McCarthy, KS & McNamara, DS 2017, 'Combining machine learning and natural language processing to assess literary text comprehension', Paper presented at 10th International Conference on Educational Data Mining, EDM 2017, Wuhan, China, 6/25/17 - 6/28/17 pp. 244-249.
Balyan R, McCarthy KS, McNamara DS. Combining machine learning and natural language processing to assess literary text comprehension. 2017. Paper presented at 10th International Conference on Educational Data Mining, EDM 2017, Wuhan, China.
Balyan, Renu ; McCarthy, Kathryn S. ; McNamara, Danielle S. / Combining machine learning and natural language processing to assess literary text comprehension. Paper presented at 10th International Conference on Educational Data Mining, EDM 2017, Wuhan, China.6 p.
@conference{3de3f6d5325e4665a0a97d65046500e6,
title = "Combining machine learning and natural language processing to assess literary text comprehension",
abstract = "This study examined how machine learning and natural language processing (NLP) techniques can be leveraged to assess the interpretive behavior that is required for successful literary text comprehension. We compared the accuracy of seven different machine learning classification algorithms in predicting human ratings of student essays about literary works. Three types of NLP feature sets: unigrams (single content words), elaborative (new) n-grams, and linguistic features were used to classify idea units (paraphrase, text-based inference, interpretive inference). The most accurate classifications emerged using all three NLP features sets in combination, with accuracy ranging from 0.61 to 0.94 (F=0.18 to 0.81). Random Forests, which employs multiple decision trees and a bagging approach, was the most accurate classifier for these data. In contrast, the single classifier, Trees, which tends to “overfit” the data during training, was the least accurate. Ensemble classifiers were generally more accurate than single classifiers. However, Support Vector Machines accuracy was comparable to that of the ensemble classifiers. This is likely due to Support Vector Machines’ unique ability to support high dimension feature spaces. The findings suggest that combining the power of NLP and machine learning is an effective means of automating literary text comprehension assessment.",
keywords = "Classification, Interpretation, Natural language processing, Supervised machine learning",
author = "Renu Balyan and McCarthy, {Kathryn S.} and McNamara, {Danielle S.}",
year = "2017",
month = "1",
day = "1",
language = "English (US)",
pages = "244--249",
note = "10th International Conference on Educational Data Mining, EDM 2017 ; Conference date: 25-06-2017 Through 28-06-2017",

}

TY - CONF

T1 - Combining machine learning and natural language processing to assess literary text comprehension

AU - Balyan, Renu

AU - McCarthy, Kathryn S.

AU - McNamara, Danielle S.

PY - 2017/1/1

Y1 - 2017/1/1

N2 - This study examined how machine learning and natural language processing (NLP) techniques can be leveraged to assess the interpretive behavior that is required for successful literary text comprehension. We compared the accuracy of seven different machine learning classification algorithms in predicting human ratings of student essays about literary works. Three types of NLP feature sets: unigrams (single content words), elaborative (new) n-grams, and linguistic features were used to classify idea units (paraphrase, text-based inference, interpretive inference). The most accurate classifications emerged using all three NLP features sets in combination, with accuracy ranging from 0.61 to 0.94 (F=0.18 to 0.81). Random Forests, which employs multiple decision trees and a bagging approach, was the most accurate classifier for these data. In contrast, the single classifier, Trees, which tends to “overfit” the data during training, was the least accurate. Ensemble classifiers were generally more accurate than single classifiers. However, Support Vector Machines accuracy was comparable to that of the ensemble classifiers. This is likely due to Support Vector Machines’ unique ability to support high dimension feature spaces. The findings suggest that combining the power of NLP and machine learning is an effective means of automating literary text comprehension assessment.

AB - This study examined how machine learning and natural language processing (NLP) techniques can be leveraged to assess the interpretive behavior that is required for successful literary text comprehension. We compared the accuracy of seven different machine learning classification algorithms in predicting human ratings of student essays about literary works. Three types of NLP feature sets: unigrams (single content words), elaborative (new) n-grams, and linguistic features were used to classify idea units (paraphrase, text-based inference, interpretive inference). The most accurate classifications emerged using all three NLP features sets in combination, with accuracy ranging from 0.61 to 0.94 (F=0.18 to 0.81). Random Forests, which employs multiple decision trees and a bagging approach, was the most accurate classifier for these data. In contrast, the single classifier, Trees, which tends to “overfit” the data during training, was the least accurate. Ensemble classifiers were generally more accurate than single classifiers. However, Support Vector Machines accuracy was comparable to that of the ensemble classifiers. This is likely due to Support Vector Machines’ unique ability to support high dimension feature spaces. The findings suggest that combining the power of NLP and machine learning is an effective means of automating literary text comprehension assessment.

KW - Classification

KW - Interpretation

KW - Natural language processing

KW - Supervised machine learning

UR - http://www.scopus.com/inward/record.url?scp=85061969014&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061969014&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85061969014

SP - 244

EP - 249

ER -