Development and initial validation of the engineering design metacognitive questionnaire

Metacognition is the process of thinking about thinking, which refers to students' ability to control cognition to ensure that learning goals are achieved or a problem is solved. It is a complex process that depends on and influences students' understanding about themselves as thinkers and learners, and usually precedes and follows cognitive activity. Metacognitive skill plays a particularly critical role in real-life or open-ended tasks, such as solving ill-structured design problems. While there is growing interest in metacognitive research, few assessment tools have been developed in the context of engineering design, particularly within classroom environments. The objective of the present paper is to discuss the process of Engineering Design and Metacognitive Questionnaire (EDMQ) development and initial validation, specifically the process of face and content validity. The instrument development is grounded in Butler and Cartier's self-regulated learning (SRL) model which describes the interplay between motivation, cognition, and metacognition within academic activities such as design. The questionnaire is adapted from their works include the Inquiry Learning Questionnaire and the Learning through Reading Questionnaire. A rubric matrix combined Butler and Cartier's SRL features and the Dym and Little's design process and team management components was used in the instrument development. Dym and Little contended that the design process consists of five phases: problem definition, conceptual design, preliminary design, detailed design, and design communication. The EDMQ include items that address cognitive strategies both in design process and team management activities. Three subsections of the EDMQ were designed to capture students' perception of metacognition at the early, middle, and final stages of the design task across design processes, respectively; the first subsection of EDMQ captures task interpretation and planning strategies; the second subsection captures cognitive actions and monitoring and fix-up strategies; the third subsection captures students' judgment of their design outcomes. Six undergraduate engineering students were invited in the face validity process. Moreover, the content validity involved two engineering professors and two experts in self-regulated learning. The resulting survey instrument contains 127 questionnaire items assessing five SRL features: Task interpretation, planning strategies, cognitive actions, monitoring and fix-up strategies, and criteria of success. This survey instrument may be useful for cognitive and metacognitive research and assessing design processes in the context of engineering design project.