A Systematic Approach to Evaluating Design Prompts in Supporting Experimental Design Research
Editor: Wartzack, Sandro; Schleich, Benjamin; Gon
Author: Patel, Apurva; Elena, Maria-Vittoria; Summers, Joshua
Institution: Clemson University
Section: Methods of design research
DOI number: https://doi.org/10.1017/dsi.2019.282
Experiments that study engineering behavior in design often rely on participants responding to a given design prompt or a problem statement. Moreover, researchers often find themselves testing multiple variables with a relatively small participant pool. In such situations multiple design prompts may be used to boost replication by giving each participant an equivalent problem with a different experimental condition. This paper presents a systematic approach to compare given design prompts using a two-step process that allows an initial comparison of the prompts and a post-experiment verification of the similarity of the given prompts. Comparison metrics are provided which can be used to evaluate a level of similarity of existing prompts as well as develop similar problems. These metrics include complexity (size, coupling, and solvability), familiarity, and prompt structure. Statistical methods are discussed for post-experiment verification. Guidelines are provided for a post-experiment survey which may be used for an additional perspective of prompt similarity. The proposed approach is demonstrated using an experiment where two design prompts were used for within-subject replication.