A simulation based estimation of crowd ability and its influence on crowdsourced evaluation of design concepts

Alex Burnap, Yi Ren, Panos Y. Papalambros, Richard Gonzalez, Richard Gerth

Research output: Chapter in Book/Report/Conference proceedingConference contribution

12 Citations (Scopus)

Abstract

Crowdsourced evaluation is a promising method for evaluating attributes of design concepts that require human input. One factor in obtaining good evaluations is the ratio of high-ability to low-ability participants within the crowd. In this paper we introduce a Bayesian network model capable of finding participants with high design evaluation ability, so that their evaluations may be weighted more than those of the rest of the crowd. The Bayesian network model also estimates a score of how well each design concept performs with respect to a design attribute without knowledge of the true scores. Monte Carlo simulation studies tested the quality of the estimations on a variety of crowds consisting of participants with different evaluation ability. Results suggest that the Bayesian network model estimates design attribute performance scores much closer to their true values than simply weighting the evaluations from all participants in the crowd equally. This finding holds true even when the group of high ability participants is a small percentage of the entire crowd.

Original languageEnglish (US)
Title of host publication39th Design Automation Conference
PublisherAmerican Society of Mechanical Engineers
Volume3 B
ISBN (Print)9780791855898
DOIs
StatePublished - 2013
Externally publishedYes
EventASME 2013 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE 2013 - Portland, OR, United States
Duration: Aug 4 2013Aug 7 2013

Other

OtherASME 2013 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE 2013
CountryUnited States
CityPortland, OR
Period8/4/138/7/13

Fingerprint

Bayesian networks
Evaluation
Bayesian Model
Bayesian Networks
Network Model
Simulation
Attribute
Estimate
Weighting
Percentage
Influence
Concepts
Design
Monte Carlo Simulation
Simulation Study
Entire

Keywords

  • Crowdsourcing
  • Design concept evaluation
  • Machine learning

ASJC Scopus subject areas

  • Mechanical Engineering
  • Computer Graphics and Computer-Aided Design
  • Computer Science Applications
  • Modeling and Simulation

Cite this

Burnap, A., Ren, Y., Papalambros, P. Y., Gonzalez, R., & Gerth, R. (2013). A simulation based estimation of crowd ability and its influence on crowdsourced evaluation of design concepts. In 39th Design Automation Conference (Vol. 3 B). [V03BT03A004] American Society of Mechanical Engineers. https://doi.org/10.1115/DETC2013-13020

A simulation based estimation of crowd ability and its influence on crowdsourced evaluation of design concepts. / Burnap, Alex; Ren, Yi; Papalambros, Panos Y.; Gonzalez, Richard; Gerth, Richard.

39th Design Automation Conference. Vol. 3 B American Society of Mechanical Engineers, 2013. V03BT03A004.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Burnap, A, Ren, Y, Papalambros, PY, Gonzalez, R & Gerth, R 2013, A simulation based estimation of crowd ability and its influence on crowdsourced evaluation of design concepts. in 39th Design Automation Conference. vol. 3 B, V03BT03A004, American Society of Mechanical Engineers, ASME 2013 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE 2013, Portland, OR, United States, 8/4/13. https://doi.org/10.1115/DETC2013-13020
Burnap A, Ren Y, Papalambros PY, Gonzalez R, Gerth R. A simulation based estimation of crowd ability and its influence on crowdsourced evaluation of design concepts. In 39th Design Automation Conference. Vol. 3 B. American Society of Mechanical Engineers. 2013. V03BT03A004 https://doi.org/10.1115/DETC2013-13020
Burnap, Alex ; Ren, Yi ; Papalambros, Panos Y. ; Gonzalez, Richard ; Gerth, Richard. / A simulation based estimation of crowd ability and its influence on crowdsourced evaluation of design concepts. 39th Design Automation Conference. Vol. 3 B American Society of Mechanical Engineers, 2013.
@inproceedings{c8ab07e130b44586b9183151df38dfb4,
title = "A simulation based estimation of crowd ability and its influence on crowdsourced evaluation of design concepts",
abstract = "Crowdsourced evaluation is a promising method for evaluating attributes of design concepts that require human input. One factor in obtaining good evaluations is the ratio of high-ability to low-ability participants within the crowd. In this paper we introduce a Bayesian network model capable of finding participants with high design evaluation ability, so that their evaluations may be weighted more than those of the rest of the crowd. The Bayesian network model also estimates a score of how well each design concept performs with respect to a design attribute without knowledge of the true scores. Monte Carlo simulation studies tested the quality of the estimations on a variety of crowds consisting of participants with different evaluation ability. Results suggest that the Bayesian network model estimates design attribute performance scores much closer to their true values than simply weighting the evaluations from all participants in the crowd equally. This finding holds true even when the group of high ability participants is a small percentage of the entire crowd.",
keywords = "Crowdsourcing, Design concept evaluation, Machine learning",
author = "Alex Burnap and Yi Ren and Papalambros, {Panos Y.} and Richard Gonzalez and Richard Gerth",
year = "2013",
doi = "10.1115/DETC2013-13020",
language = "English (US)",
isbn = "9780791855898",
volume = "3 B",
booktitle = "39th Design Automation Conference",
publisher = "American Society of Mechanical Engineers",

}

TY - GEN

T1 - A simulation based estimation of crowd ability and its influence on crowdsourced evaluation of design concepts

AU - Burnap, Alex

AU - Ren, Yi

AU - Papalambros, Panos Y.

AU - Gonzalez, Richard

AU - Gerth, Richard

PY - 2013

Y1 - 2013

N2 - Crowdsourced evaluation is a promising method for evaluating attributes of design concepts that require human input. One factor in obtaining good evaluations is the ratio of high-ability to low-ability participants within the crowd. In this paper we introduce a Bayesian network model capable of finding participants with high design evaluation ability, so that their evaluations may be weighted more than those of the rest of the crowd. The Bayesian network model also estimates a score of how well each design concept performs with respect to a design attribute without knowledge of the true scores. Monte Carlo simulation studies tested the quality of the estimations on a variety of crowds consisting of participants with different evaluation ability. Results suggest that the Bayesian network model estimates design attribute performance scores much closer to their true values than simply weighting the evaluations from all participants in the crowd equally. This finding holds true even when the group of high ability participants is a small percentage of the entire crowd.

AB - Crowdsourced evaluation is a promising method for evaluating attributes of design concepts that require human input. One factor in obtaining good evaluations is the ratio of high-ability to low-ability participants within the crowd. In this paper we introduce a Bayesian network model capable of finding participants with high design evaluation ability, so that their evaluations may be weighted more than those of the rest of the crowd. The Bayesian network model also estimates a score of how well each design concept performs with respect to a design attribute without knowledge of the true scores. Monte Carlo simulation studies tested the quality of the estimations on a variety of crowds consisting of participants with different evaluation ability. Results suggest that the Bayesian network model estimates design attribute performance scores much closer to their true values than simply weighting the evaluations from all participants in the crowd equally. This finding holds true even when the group of high ability participants is a small percentage of the entire crowd.

KW - Crowdsourcing

KW - Design concept evaluation

KW - Machine learning

UR - http://www.scopus.com/inward/record.url?scp=84897006005&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84897006005&partnerID=8YFLogxK

U2 - 10.1115/DETC2013-13020

DO - 10.1115/DETC2013-13020

M3 - Conference contribution

SN - 9780791855898

VL - 3 B

BT - 39th Design Automation Conference

PB - American Society of Mechanical Engineers

ER -