Intention-based scoring: An approach to measuring success at solving the composition problem

H. Chad Lane, Kurt VanLehn

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Scopus citations

Abstract

Traditional methods of evaluating student programs are not always appropriate for assessment of different instructional interventions. They tend to focus on the final product rather than on the process that led to it. This paper presents intention-based scoring (IBS), an approach to measuring programming ability that requires inspection of intermediate programs produced over the course of an implementation rather than just the one at the end. The intent is to assess a student's ability to produce algorithmically correct code on the first attempt at achieving each program goal. In other words, the goal is to answer question "How close was the student to being initially correct?" rather than the the ability to ultimately produce a working program. To produce an IBS, it is necessary to inspect a student's online protocol, which is defined as the collection of all programs submitted to a compiler. IBS involves a three-phase process of (1) identification of the subset of all programs in a protocol that represent the initial attempts at achieving programming goals, (2) analysis of the bugs in those programs, and (3) rubric-based scoring of the resulting tagged programs. We conclude with an example application of IBS in the evaluation of a tutoring system for beginning programmers and also show how an IBS can be broken down by the underlying bug categories to reveal more subtle differences.

Original languageEnglish (US)
Title of host publicationProceedings of the Thirty-Sixth SIGCSE Technical Symposium on Computer Science Education, SIGCSE 2005
Pages373-377
Number of pages5
StatePublished - Jun 27 2005
Externally publishedYes
EventProceedings of the Thirty-Sixth SIGCSE Technical Symposium on Computer Science Education, SIGCSE 2005 - St. Louis, MO, United States
Duration: Feb 23 2005Feb 27 2005

Publication series

NameProceedings of the Thirty-Sixth SIGCSE Technical Symposium on Computer Science Education, SIGCSE 2005

Conference

ConferenceProceedings of the Thirty-Sixth SIGCSE Technical Symposium on Computer Science Education, SIGCSE 2005
Country/TerritoryUnited States
CitySt. Louis, MO
Period2/23/052/27/05

Keywords

  • Intelligent tutoring systems
  • Intention-based scoring
  • Novice programming
  • Online protocols
  • Structured programming

ASJC Scopus subject areas

  • Engineering(all)

Fingerprint

Dive into the research topics of 'Intention-based scoring: An approach to measuring success at solving the composition problem'. Together they form a unique fingerprint.

Cite this