TY - GEN
T1 - Predicting question quality using recurrent neural networks
AU - Ruseti, Stefan
AU - Dascalu, Mihai
AU - Johnson, Amy
AU - Balyan, Renu
AU - Kopp, Kristopher J.
AU - McNamara, Danielle
AU - Crossley, Scott A.
AU - Trausan-Matu, Stefan
N1 - Funding Information:
Acknowledgments. This research was partially supported by the 644187 EC H2020 Realising an Applied Gaming Eco-system (RAGE) project, the FP7 2008-212578 LTfLL project, the Department of Education, Institute of Education Sciences - Grant R305A130124, as well as the Department of Defense, Office of Naval Research - Grants N00014140343 and N000141712300.
Funding Information:
This research was partially supported by the 644187 EC H2020 Realising an Applied Gaming Eco-system (RAGE) project, the FP7 2008-212578 LTfLL project, the Department of Education, Institute of Education Sciences-Grant R305A130124, as well as the Department of Defense, Office of Naval Research-Grants N00014140343 and N000141712300.
Publisher Copyright:
© Springer International Publishing AG, part of Springer Nature 2018.
PY - 2018
Y1 - 2018
N2 - This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this dataset based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22%, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6%). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.
AB - This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this dataset based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22%, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6%). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.
KW - Question asking
KW - Recurrent neural network
KW - Word embeddings
UR - http://www.scopus.com/inward/record.url?scp=85049368697&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85049368697&partnerID=8YFLogxK
U2 - 10.1007/978-3-319-93843-1_36
DO - 10.1007/978-3-319-93843-1_36
M3 - Conference contribution
AN - SCOPUS:85049368697
SN - 9783319938424
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 491
EP - 502
BT - Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings
A2 - Mavrikis, Manolis
A2 - Penstein Rosé, Carolyn
A2 - McLaren, Bruce
A2 - Hoppe, H. Ulrich
A2 - Luckin, Rose
A2 - Porayska-Pomsta, Kaska
A2 - du Boulay, Benedict
A2 - Martinez-Maldonado, Roberto
PB - Springer Verlag
T2 - 19th International Conference on Artificial Intelligence in Education, AIED 2018
Y2 - 27 June 2018 through 30 June 2018
ER -