Bring it on! Challenges encountered while building a comprehensive tutoring system using ReaderBench

Marilena Panaite, Mihai Dascalu, Amy Johnson, Renu Balyan, Jianmin Dai, Danielle McNamara, Stefan Trausan-Matu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students’ practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different self-explanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59% (adjacent accuracy = 97%; Kappa =.43).

Original languageEnglish (US)
Title of host publicationArtificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings
PublisherSpringer Verlag
Pages409-419
Number of pages11
ISBN (Print)9783319938424
DOIs
StatePublished - Jan 1 2018
Event19th International Conference on Artificial Intelligence in Education, AIED 2018 - London, United Kingdom
Duration: Jun 27 2018Jun 30 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10947 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other19th International Conference on Artificial Intelligence in Education, AIED 2018
CountryUnited Kingdom
CityLondon
Period6/27/186/30/18

Fingerprint

Intelligent Tutoring Systems
Virtual Machine
Intelligent systems
Rule-based Systems
Hyperparameters
Knowledge based systems
Processing
Scoring
Natural Language
Learning algorithms
Work Flow
Preprocessing
Learning systems
Learning Algorithm
Machine Learning
Adjacent
Students
Grid
Feedback
Evaluation

Keywords

  • Intelligent tutoring systems
  • Natural language processing
  • ReaderBench
  • Self-explanations
  • Support vector machines

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Panaite, M., Dascalu, M., Johnson, A., Balyan, R., Dai, J., McNamara, D., & Trausan-Matu, S. (2018). Bring it on! Challenges encountered while building a comprehensive tutoring system using ReaderBench. In Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings (pp. 409-419). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10947 LNAI). Springer Verlag. https://doi.org/10.1007/978-3-319-93843-1_30

Bring it on! Challenges encountered while building a comprehensive tutoring system using ReaderBench. / Panaite, Marilena; Dascalu, Mihai; Johnson, Amy; Balyan, Renu; Dai, Jianmin; McNamara, Danielle; Trausan-Matu, Stefan.

Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings. Springer Verlag, 2018. p. 409-419 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10947 LNAI).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Panaite, M, Dascalu, M, Johnson, A, Balyan, R, Dai, J, McNamara, D & Trausan-Matu, S 2018, Bring it on! Challenges encountered while building a comprehensive tutoring system using ReaderBench. in Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10947 LNAI, Springer Verlag, pp. 409-419, 19th International Conference on Artificial Intelligence in Education, AIED 2018, London, United Kingdom, 6/27/18. https://doi.org/10.1007/978-3-319-93843-1_30
Panaite M, Dascalu M, Johnson A, Balyan R, Dai J, McNamara D et al. Bring it on! Challenges encountered while building a comprehensive tutoring system using ReaderBench. In Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings. Springer Verlag. 2018. p. 409-419. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-93843-1_30
Panaite, Marilena ; Dascalu, Mihai ; Johnson, Amy ; Balyan, Renu ; Dai, Jianmin ; McNamara, Danielle ; Trausan-Matu, Stefan. / Bring it on! Challenges encountered while building a comprehensive tutoring system using ReaderBench. Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings. Springer Verlag, 2018. pp. 409-419 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{eb71db1513244e128611484e4dc983d6,
title = "Bring it on! Challenges encountered while building a comprehensive tutoring system using ReaderBench",
abstract = "Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students’ practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different self-explanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59{\%} (adjacent accuracy = 97{\%}; Kappa =.43).",
keywords = "Intelligent tutoring systems, Natural language processing, ReaderBench, Self-explanations, Support vector machines",
author = "Marilena Panaite and Mihai Dascalu and Amy Johnson and Renu Balyan and Jianmin Dai and Danielle McNamara and Stefan Trausan-Matu",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/978-3-319-93843-1_30",
language = "English (US)",
isbn = "9783319938424",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "409--419",
booktitle = "Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings",

}

TY - GEN

T1 - Bring it on! Challenges encountered while building a comprehensive tutoring system using ReaderBench

AU - Panaite, Marilena

AU - Dascalu, Mihai

AU - Johnson, Amy

AU - Balyan, Renu

AU - Dai, Jianmin

AU - McNamara, Danielle

AU - Trausan-Matu, Stefan

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students’ practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different self-explanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59% (adjacent accuracy = 97%; Kappa =.43).

AB - Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students’ practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different self-explanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59% (adjacent accuracy = 97%; Kappa =.43).

KW - Intelligent tutoring systems

KW - Natural language processing

KW - ReaderBench

KW - Self-explanations

KW - Support vector machines

UR - http://www.scopus.com/inward/record.url?scp=85049369683&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85049369683&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-93843-1_30

DO - 10.1007/978-3-319-93843-1_30

M3 - Conference contribution

SN - 9783319938424

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 409

EP - 419

BT - Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings

PB - Springer Verlag

ER -