Predicting question quality using recurrent neural networks

Stefan Ruseti, Mihai Dascalu, Amy Johnson, Renu Balyan, Kristopher J. Kopp, Danielle McNamara, Scott A. Crossley, Stefan Trausan-Matu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this dataset based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22%, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6%). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.

Original languageEnglish (US)
Title of host publicationArtificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings
PublisherSpringer Verlag
Pages491-502
Number of pages12
ISBN (Print)9783319938424
DOIs
StatePublished - Jan 1 2018
Event19th International Conference on Artificial Intelligence in Education, AIED 2018 - London, United Kingdom
Duration: Jun 27 2018Jun 30 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10947 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other19th International Conference on Artificial Intelligence in Education, AIED 2018
CountryUnited Kingdom
CityLondon
Period6/27/186/30/18

Fingerprint

Recurrent neural networks
Recurrent Neural Networks
Intelligent systems
Network architecture
Learning systems
Intelligent Tutoring Systems
Learning Environment
Network Architecture
Students
Feedback
Machine Learning
Predict
Prediction
Experiments
Cell
Experiment
Model

Keywords

  • Question asking
  • Recurrent neural network
  • Word embeddings

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Ruseti, S., Dascalu, M., Johnson, A., Balyan, R., Kopp, K. J., McNamara, D., ... Trausan-Matu, S. (2018). Predicting question quality using recurrent neural networks. In Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings (pp. 491-502). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10947 LNAI). Springer Verlag. https://doi.org/10.1007/978-3-319-93843-1_36

Predicting question quality using recurrent neural networks. / Ruseti, Stefan; Dascalu, Mihai; Johnson, Amy; Balyan, Renu; Kopp, Kristopher J.; McNamara, Danielle; Crossley, Scott A.; Trausan-Matu, Stefan.

Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings. Springer Verlag, 2018. p. 491-502 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10947 LNAI).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Ruseti, S, Dascalu, M, Johnson, A, Balyan, R, Kopp, KJ, McNamara, D, Crossley, SA & Trausan-Matu, S 2018, Predicting question quality using recurrent neural networks. in Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10947 LNAI, Springer Verlag, pp. 491-502, 19th International Conference on Artificial Intelligence in Education, AIED 2018, London, United Kingdom, 6/27/18. https://doi.org/10.1007/978-3-319-93843-1_36
Ruseti S, Dascalu M, Johnson A, Balyan R, Kopp KJ, McNamara D et al. Predicting question quality using recurrent neural networks. In Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings. Springer Verlag. 2018. p. 491-502. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-93843-1_36
Ruseti, Stefan ; Dascalu, Mihai ; Johnson, Amy ; Balyan, Renu ; Kopp, Kristopher J. ; McNamara, Danielle ; Crossley, Scott A. ; Trausan-Matu, Stefan. / Predicting question quality using recurrent neural networks. Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings. Springer Verlag, 2018. pp. 491-502 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{01ac1c9ef344428db81852dd767c7d70,
title = "Predicting question quality using recurrent neural networks",
abstract = "This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this dataset based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22{\%}, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6{\%}). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.",
keywords = "Question asking, Recurrent neural network, Word embeddings",
author = "Stefan Ruseti and Mihai Dascalu and Amy Johnson and Renu Balyan and Kopp, {Kristopher J.} and Danielle McNamara and Crossley, {Scott A.} and Stefan Trausan-Matu",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/978-3-319-93843-1_36",
language = "English (US)",
isbn = "9783319938424",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "491--502",
booktitle = "Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings",

}

TY - GEN

T1 - Predicting question quality using recurrent neural networks

AU - Ruseti, Stefan

AU - Dascalu, Mihai

AU - Johnson, Amy

AU - Balyan, Renu

AU - Kopp, Kristopher J.

AU - McNamara, Danielle

AU - Crossley, Scott A.

AU - Trausan-Matu, Stefan

PY - 2018/1/1

Y1 - 2018/1/1

N2 - This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this dataset based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22%, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6%). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.

AB - This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this dataset based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22%, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6%). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.

KW - Question asking

KW - Recurrent neural network

KW - Word embeddings

UR - http://www.scopus.com/inward/record.url?scp=85049368697&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85049368697&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-93843-1_36

DO - 10.1007/978-3-319-93843-1_36

M3 - Conference contribution

SN - 9783319938424

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 491

EP - 502

BT - Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings

PB - Springer Verlag

ER -