A simulation based estimation of crowd ability and its influence on crowdsourced evaluation of design concepts

Alex Burnap, Yi Ren, Panos Y. Papalambros, Richard Gonzalez, Richard Gerth

Research output: Chapter in Book/Report/Conference proceedingConference contribution

16 Scopus citations

Abstract

Crowdsourced evaluation is a promising method for evaluating attributes of design concepts that require human input. One factor in obtaining good evaluations is the ratio of high-ability to low-ability participants within the crowd. In this paper we introduce a Bayesian network model capable of finding participants with high design evaluation ability, so that their evaluations may be weighted more than those of the rest of the crowd. The Bayesian network model also estimates a score of how well each design concept performs with respect to a design attribute without knowledge of the true scores. Monte Carlo simulation studies tested the quality of the estimations on a variety of crowds consisting of participants with different evaluation ability. Results suggest that the Bayesian network model estimates design attribute performance scores much closer to their true values than simply weighting the evaluations from all participants in the crowd equally. This finding holds true even when the group of high ability participants is a small percentage of the entire crowd.

Original languageEnglish (US)
Title of host publication39th Design Automation Conference
PublisherAmerican Society of Mechanical Engineers
ISBN (Print)9780791855898
DOIs
StatePublished - Jan 1 2013
Externally publishedYes
EventASME 2013 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE 2013 - Portland, OR, United States
Duration: Aug 4 2013Aug 7 2013

Publication series

NameProceedings of the ASME Design Engineering Technical Conference
Volume3 B

Other

OtherASME 2013 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE 2013
CountryUnited States
CityPortland, OR
Period8/4/138/7/13

Keywords

  • Crowdsourcing
  • Design concept evaluation
  • Machine learning

ASJC Scopus subject areas

  • Modeling and Simulation
  • Mechanical Engineering
  • Computer Science Applications
  • Computer Graphics and Computer-Aided Design

Fingerprint Dive into the research topics of 'A simulation based estimation of crowd ability and its influence on crowdsourced evaluation of design concepts'. Together they form a unique fingerprint.

  • Cite this

    Burnap, A., Ren, Y., Papalambros, P. Y., Gonzalez, R., & Gerth, R. (2013). A simulation based estimation of crowd ability and its influence on crowdsourced evaluation of design concepts. In 39th Design Automation Conference [V03BT03A004] (Proceedings of the ASME Design Engineering Technical Conference; Vol. 3 B). American Society of Mechanical Engineers. https://doi.org/10.1115/DETC2013-13020