Mental models of metal models: a comparison of mental model measurement techniques

Although use of the mental model construct has proliferated in recent research, the construct lacks a clear definition and an agreed upon method of measurement. Furthermore, the reliability and validity of the different measurement techniques in use have not been established, thereby making generalizations across studies of mental models difficult. The purpose of the current project was to assess several methods of measuring mental models in terms of their reliability/stability over time. Subjects' mental models of the automobile engine system were elicited on two occasions separated by one week, using seven different knowledge elicitation techniques. Subjects' level of experience was also measured to allow comparisons between experts and novices. The results indicate that each of the measurement techniques tended to be reliable for both experts and novices. However, reliability tended to be greater for experts than novices. Additionally, experts tended to agree with each other more than did the novices. Some evidence also indicated that the results from the similarity ratings and subsequent Pathfinder analysis converged with those from the structured interviews.