Developing a method to measure the metacognitive effects of a course on design, engineering and technology over time

Measuring and tracking how individuals become aware of their own understanding (metacognition) cannot easily be measured by traditional tests or assessments. Consequently, this paper presents the development and application of a rubric to examine qualitative data that illustrates how graduate students in science education, who were enrolled in a Design, Engineering and Technology. (DET) course, became aware of the changes in their understanding of DET. Weekly reflection papers, weekly written pre and post tests and lesson plans were used as data sources. A rubric linking the course outcomes with six major categories (engineering as a design process, gender and diversity, societal relevance of engineering, technical self-efficacy, tinkering self-efficacy and transfer to classroom teaching) was developed to code text. Several passes through the data led to the refinements for the six categories that allowed the coding of almost all of the text. We specifically looked for shifts in understanding over a 15-week period and an awareness that these shifts were taking place (e.g. "It's not that I had a bad attitude about technology to begin with, rather this class as a whole and our group project has forced me to think about its appropriate applications at the K-12 level." and "Both technology education papers addressed the difference between technology education and educational technology - two different concepts I had not thought of before"). Our technique allowed us to capture the subtleties of understanding and the progression of metacognition. The rubric demonstrated that the DET course had a strong impact on students thinking about and applying DET to teaching.