Assessing Entailer with a corpus of natural language from an intelligent tutoring system

Philip M. McCarthy, Vasile Rus, Scott A. Crossley, Sarah C. Bigham, Arthur C. Graesser, Danielle S. McNamara

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Scopus citations

Abstract

In this study, we compared Entailer, a computational tool that evaluates the degree to which one text is entailed by another, to a variety of other text relatedness metrics (LSA, lemma overlap, and MED). Our corpus was a subset of 100 self-explanations of sentences from a recent experiment on interactions between students and iSTART, an Intelligent Tutoring System that helps students to apply metacognitive strategies to enhance deep comprehension. The sentence pairs were hand coded by experts in discourse processing across four categories of text relatedness: entailment, implicature, elaboration, and paraphrase. A series of regression analyses revealed that Entailer was the best measure for approximating these hand coded values. The Entailer could explain approximately 50% of the variance for entailment, 38% of the variance for elaboration, and 23% of the variance for paraphrase. LSA contributed marginally to the entailment model. Neither lemma-overlap nor MED contributed to any of the models, although a modified version of MED did correlate significantly with both the entailment and paraphrase hand coded evaluations. This study is an important step towards developing a set of indices designed to better assess natural language input by students in Intelligent Tutoring Systems.

Original languageEnglish (US)
Title of host publicationProceedings of the Twentieth International Florida Artificial Intelligence Research Society Conference, FLAIRS 2007
Pages247-252
Number of pages6
StatePublished - Dec 28 2007
Event20th International Florida Artificial Intelligence Research Society Conference, FLAIRS 2007 - Key West, FL, United States
Duration: May 7 2007May 9 2007

Publication series

NameProceedings of the Twentieth International Florida Artificial Intelligence Research Society Conference, FLAIRS 2007

Other

Other20th International Florida Artificial Intelligence Research Society Conference, FLAIRS 2007
CountryUnited States
CityKey West, FL
Period5/7/075/9/07

    Fingerprint

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software

Cite this

McCarthy, P. M., Rus, V., Crossley, S. A., Bigham, S. C., Graesser, A. C., & McNamara, D. S. (2007). Assessing Entailer with a corpus of natural language from an intelligent tutoring system. In Proceedings of the Twentieth International Florida Artificial Intelligence Research Society Conference, FLAIRS 2007 (pp. 247-252). (Proceedings of the Twentieth International Florida Artificial Intelligence Research Society Conference, FLAIRS 2007).