Metacognitive Overload!

Positive and Negative Effects of Metacognitive Prompts in an Intelligent Tutoring System

Kathryn S. McCarthy, Aaron D. Likens, Amy Johnson, Tricia A. Guerrero, Danielle McNamara

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Research suggests that promoting metacognitive awareness can increase performance in, and learning from, intelligent tutoring systems (ITSs). The current work examines the effects of two metacognitive prompts within iSTART, a reading comprehension strategy ITS in which students practice writing quality self-explanations. In addition to comparing iSTART practice to a no-training control, those in the iSTART condition (n = 116) were randomly assigned to a 2 (performance threshold: off, on) × 2(self-assessment: off, on) design. The performance threshold notified students when their average self-explanation score was below an experimenter-set threshold and the self-assessment prompted students to estimate their self-explanation score on the current trial. Students who practiced with iSTART had higher posttest self-explanation scores and inference comprehension scores on a transfer test than students in the no training control, replicating previous benefits for iSTART. However, there were no effects of either metacognitive prompt on these learning outcomes. In-system self-explanation scores indicated that the metacognitive prompts were detrimental to performance relative to standard iSTART practice. This study did not find benefits of metacognitive prompts in enhancing performance during practice or after the completion of training. Such findings support the idea that improving reading comprehension strategies comes from deliberate practice with actionable feedback rather than explicit metacognitive supports.

Original languageEnglish (US)
Pages (from-to)420-438
Number of pages19
JournalInternational Journal of Artificial Intelligence in Education
Volume28
Issue number3
DOIs
StatePublished - Sep 1 2018

Fingerprint

Intelligent systems
Students
comprehension
performance
self-assessment
student
learning
Feedback

Keywords

  • Intelligent tutoring systems
  • Log data
  • Metacognition
  • Reading comprehension

ASJC Scopus subject areas

  • Education
  • Computational Theory and Mathematics

Cite this

Metacognitive Overload! Positive and Negative Effects of Metacognitive Prompts in an Intelligent Tutoring System. / McCarthy, Kathryn S.; Likens, Aaron D.; Johnson, Amy; Guerrero, Tricia A.; McNamara, Danielle.

In: International Journal of Artificial Intelligence in Education, Vol. 28, No. 3, 01.09.2018, p. 420-438.

Research output: Contribution to journalArticle

@article{d58d1860fd404adcb4832d43357fa00f,
title = "Metacognitive Overload!: Positive and Negative Effects of Metacognitive Prompts in an Intelligent Tutoring System",
abstract = "Research suggests that promoting metacognitive awareness can increase performance in, and learning from, intelligent tutoring systems (ITSs). The current work examines the effects of two metacognitive prompts within iSTART, a reading comprehension strategy ITS in which students practice writing quality self-explanations. In addition to comparing iSTART practice to a no-training control, those in the iSTART condition (n = 116) were randomly assigned to a 2 (performance threshold: off, on) × 2(self-assessment: off, on) design. The performance threshold notified students when their average self-explanation score was below an experimenter-set threshold and the self-assessment prompted students to estimate their self-explanation score on the current trial. Students who practiced with iSTART had higher posttest self-explanation scores and inference comprehension scores on a transfer test than students in the no training control, replicating previous benefits for iSTART. However, there were no effects of either metacognitive prompt on these learning outcomes. In-system self-explanation scores indicated that the metacognitive prompts were detrimental to performance relative to standard iSTART practice. This study did not find benefits of metacognitive prompts in enhancing performance during practice or after the completion of training. Such findings support the idea that improving reading comprehension strategies comes from deliberate practice with actionable feedback rather than explicit metacognitive supports.",
keywords = "Intelligent tutoring systems, Log data, Metacognition, Reading comprehension",
author = "McCarthy, {Kathryn S.} and Likens, {Aaron D.} and Amy Johnson and Guerrero, {Tricia A.} and Danielle McNamara",
year = "2018",
month = "9",
day = "1",
doi = "10.1007/s40593-018-0164-5",
language = "English (US)",
volume = "28",
pages = "420--438",
journal = "International Journal of Artificial Intelligence in Education",
issn = "1560-4292",
publisher = "IOS Press",
number = "3",

}

TY - JOUR

T1 - Metacognitive Overload!

T2 - Positive and Negative Effects of Metacognitive Prompts in an Intelligent Tutoring System

AU - McCarthy, Kathryn S.

AU - Likens, Aaron D.

AU - Johnson, Amy

AU - Guerrero, Tricia A.

AU - McNamara, Danielle

PY - 2018/9/1

Y1 - 2018/9/1

N2 - Research suggests that promoting metacognitive awareness can increase performance in, and learning from, intelligent tutoring systems (ITSs). The current work examines the effects of two metacognitive prompts within iSTART, a reading comprehension strategy ITS in which students practice writing quality self-explanations. In addition to comparing iSTART practice to a no-training control, those in the iSTART condition (n = 116) were randomly assigned to a 2 (performance threshold: off, on) × 2(self-assessment: off, on) design. The performance threshold notified students when their average self-explanation score was below an experimenter-set threshold and the self-assessment prompted students to estimate their self-explanation score on the current trial. Students who practiced with iSTART had higher posttest self-explanation scores and inference comprehension scores on a transfer test than students in the no training control, replicating previous benefits for iSTART. However, there were no effects of either metacognitive prompt on these learning outcomes. In-system self-explanation scores indicated that the metacognitive prompts were detrimental to performance relative to standard iSTART practice. This study did not find benefits of metacognitive prompts in enhancing performance during practice or after the completion of training. Such findings support the idea that improving reading comprehension strategies comes from deliberate practice with actionable feedback rather than explicit metacognitive supports.

AB - Research suggests that promoting metacognitive awareness can increase performance in, and learning from, intelligent tutoring systems (ITSs). The current work examines the effects of two metacognitive prompts within iSTART, a reading comprehension strategy ITS in which students practice writing quality self-explanations. In addition to comparing iSTART practice to a no-training control, those in the iSTART condition (n = 116) were randomly assigned to a 2 (performance threshold: off, on) × 2(self-assessment: off, on) design. The performance threshold notified students when their average self-explanation score was below an experimenter-set threshold and the self-assessment prompted students to estimate their self-explanation score on the current trial. Students who practiced with iSTART had higher posttest self-explanation scores and inference comprehension scores on a transfer test than students in the no training control, replicating previous benefits for iSTART. However, there were no effects of either metacognitive prompt on these learning outcomes. In-system self-explanation scores indicated that the metacognitive prompts were detrimental to performance relative to standard iSTART practice. This study did not find benefits of metacognitive prompts in enhancing performance during practice or after the completion of training. Such findings support the idea that improving reading comprehension strategies comes from deliberate practice with actionable feedback rather than explicit metacognitive supports.

KW - Intelligent tutoring systems

KW - Log data

KW - Metacognition

KW - Reading comprehension

UR - http://www.scopus.com/inward/record.url?scp=85050955155&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050955155&partnerID=8YFLogxK

U2 - 10.1007/s40593-018-0164-5

DO - 10.1007/s40593-018-0164-5

M3 - Article

VL - 28

SP - 420

EP - 438

JO - International Journal of Artificial Intelligence in Education

JF - International Journal of Artificial Intelligence in Education

SN - 1560-4292

IS - 3

ER -