Developing a Measure to Capture Middle School Students’ Interpretive Understanding of Engineering Design

Abstract By emphasizing the engineering design process as a central concept of the subject area, the Next Generation Science Standards (NGSS) have created an impetus for advancing a model of K-12 engineering education that supports opportunities for students to meaningfully engage with the idea and practice of iterative design. Assessing meaningful learning in the K-12 context, however, is a challenge, as traditional engineering assessment practices are rarely able to measure or contribute to high levels of cognitive engagement. One aspect of meaningful learning is that it incorporates assessments that ask students to go beyond recalling information by rote, and instead invites them to engage in constructing deeper understanding (Mayer, 2002). Doing so requires instruments that can get at deeper levels of students’ capacity to think with, or interpret, engineering concepts rather than remembering information or applying fixed procedures to situations similar to those they encountered during instruction. This paper describes our effort to design and develop one such instrument using the theories and frameworks of preparation for future learning (Schwartz, Bransford, & Sears, 2005) and adaptive design expertise (Neeley, 2007). Starting with an analysis of the assessment items available to teachers in our partner school system, we begin by unpacking how traditional engagement with engineering concepts through assessment creates conditions that perpetuate common pitfalls in pre-college engineering education. We then describe our own process of developing test items, at first as an attempt to accommodate the principles of traditional assessment, creating at total of 17 items across three separate item pools, and increasingly gravitating toward the performance assessment paradigm, finally honing in on three performance assessment tasks. We then provide the results of a study carried out using assessment results generated by our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non-treatment (n = 241) groups, wherein the former participated in a two-week, NGSS-aligned unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach (Egan, 2005). The latter group were taught using the district’s existing engineering design curriculum. A series of non-parametric comparative analyses (independent two-group Mann-Whitney tests) found statistically significant differences in the pattern of student responses related to (1) fluency, (2) reflective ideation, and (3) figural elaboration of the performance assessment tasks. These results may shed light on ways that innovating engineering assessment can both contribute to and provide insight about students’ capacity to generate ideas, reflect, and demonstrate their understanding of engineering design. The utility of these findings are further discussed in terms of their relevance to teachers, researchers, curriculum designers, and other stakeholders.