Evaluation of idea generation methods for conceptual design: Effectiveness metrics and design of experiments

A wide range of formal methods have been devised and used for idea generation in conceptual design. Experimental evidence is needed to support claims regarding the effectiveness of these methods in promoting idea generation in engineering design. Towards that goal this paper presents a set of effectiveness metrics experimental methods, data collection and analysis techniques. Statistically based Design of Experiments (DOE) principles were used in developing the guidelines. Four classes of operating variables were considered to characterize the design problem and the environment. The effectiveness metrics proposed are based on outcome and consists of the quantity, quality, novelty, and variety of ideas generated. Two experimental approaches have been developed. In the Direct Method, the influence of the type of design problem and various parameters related to the procedure of an idea generation method is measured by using the method in its entirety. In the Indirect Method, each idea generation method is decomposed into key components and its overall effectiveness is predicted by experimentally studying the effectiveness of its components and their mutual interactions.