Evaluating mental model elicitation methods

Researchers have developed and applied a number of methods for measuring mental models. Unfortunately not only is the mental model construct ill-defined, but the basic research associated with it offers little guidance concerning the selection of a method for a particular application. In this paper a program of research is presented that is designed address this shortcoming. Specifically, the research involves a comparative evaluation of methods to measure mental models on the basis of the relationship between the method's output (i.e., the mental model) and the criterion of primary importance to the problem (e.g., task performance, user acceptance). It is assumed that a method should be selected on the basis of its ability to generate output that is predictive of the criterion of interest. It is likely that because the methods tap different aspects of a mental model, they will predict performance well on some tasks and criteria, but not others. As an example of this approach, data are presented that help to select the best method for measuring technicians' mental models of an electronics troubleshooting task.