When crowdsourcing fails

A study of expertise on crowdsourced design evaluation

Alex Burnap, Yi Ren, Richard Gerth, Giannis Papazoglou, Richard Gonzalez, Panos Y. Papalambros

Research output: Contribution to journalArticle

31 Citations (Scopus)

Abstract

Crowdsourced evaluation is a promising method of evaluating engineering design attributes that require human input. The challenge is to correctly estimate scores using a massive and diverse crowd, particularly when only a small subset of evaluators has the expertise to give correct evaluations. Since averaging evaluations across all evaluators will result in an inaccurate crowd evaluation, this paper benchmarks a crowd consensus model that aims to identify experts such that their evaluations may be given more weight. Simulation results indicate this crowd consensus model outperforms averaging when it correctly identifies experts in the crowd, under the assumption that only experts have consistent evaluations. However, empirical results from a real human crowd indicate this assumption may not hold even on a simple engineering design evaluation task, as clusters of consistently wrong evaluators are shown to exist along with the cluster of experts. This suggests that both averaging evaluations and a crowd consensus model that relies only on evaluations may not be adequate for engineering design tasks, accordingly calling for further research into methods of finding experts within the crowd.

Original languageEnglish (US)
Article number031101
JournalJournal of Mechanical Design, Transactions of the ASME
Volume137
Issue number3
DOIs
StatePublished - 2015
Externally publishedYes

Keywords

  • Crowd consensus
  • Crowdsourcing
  • Design evaluation
  • Evaluator expertise

ASJC Scopus subject areas

  • Mechanics of Materials
  • Mechanical Engineering
  • Computer Science Applications
  • Computer Graphics and Computer-Aided Design

Cite this

When crowdsourcing fails : A study of expertise on crowdsourced design evaluation. / Burnap, Alex; Ren, Yi; Gerth, Richard; Papazoglou, Giannis; Gonzalez, Richard; Papalambros, Panos Y.

In: Journal of Mechanical Design, Transactions of the ASME, Vol. 137, No. 3, 031101, 2015.

Research output: Contribution to journalArticle

Burnap, Alex ; Ren, Yi ; Gerth, Richard ; Papazoglou, Giannis ; Gonzalez, Richard ; Papalambros, Panos Y. / When crowdsourcing fails : A study of expertise on crowdsourced design evaluation. In: Journal of Mechanical Design, Transactions of the ASME. 2015 ; Vol. 137, No. 3.
@article{162c17a8fb6e491ebd8e306135f6eae9,
title = "When crowdsourcing fails: A study of expertise on crowdsourced design evaluation",
abstract = "Crowdsourced evaluation is a promising method of evaluating engineering design attributes that require human input. The challenge is to correctly estimate scores using a massive and diverse crowd, particularly when only a small subset of evaluators has the expertise to give correct evaluations. Since averaging evaluations across all evaluators will result in an inaccurate crowd evaluation, this paper benchmarks a crowd consensus model that aims to identify experts such that their evaluations may be given more weight. Simulation results indicate this crowd consensus model outperforms averaging when it correctly identifies experts in the crowd, under the assumption that only experts have consistent evaluations. However, empirical results from a real human crowd indicate this assumption may not hold even on a simple engineering design evaluation task, as clusters of consistently wrong evaluators are shown to exist along with the cluster of experts. This suggests that both averaging evaluations and a crowd consensus model that relies only on evaluations may not be adequate for engineering design tasks, accordingly calling for further research into methods of finding experts within the crowd.",
keywords = "Crowd consensus, Crowdsourcing, Design evaluation, Evaluator expertise",
author = "Alex Burnap and Yi Ren and Richard Gerth and Giannis Papazoglou and Richard Gonzalez and Papalambros, {Panos Y.}",
year = "2015",
doi = "10.1115/1.4029065",
language = "English (US)",
volume = "137",
journal = "Journal of Mechanical Design - Transactions of the ASME",
issn = "1050-0472",
publisher = "American Society of Mechanical Engineers(ASME)",
number = "3",

}

TY - JOUR

T1 - When crowdsourcing fails

T2 - A study of expertise on crowdsourced design evaluation

AU - Burnap, Alex

AU - Ren, Yi

AU - Gerth, Richard

AU - Papazoglou, Giannis

AU - Gonzalez, Richard

AU - Papalambros, Panos Y.

PY - 2015

Y1 - 2015

N2 - Crowdsourced evaluation is a promising method of evaluating engineering design attributes that require human input. The challenge is to correctly estimate scores using a massive and diverse crowd, particularly when only a small subset of evaluators has the expertise to give correct evaluations. Since averaging evaluations across all evaluators will result in an inaccurate crowd evaluation, this paper benchmarks a crowd consensus model that aims to identify experts such that their evaluations may be given more weight. Simulation results indicate this crowd consensus model outperforms averaging when it correctly identifies experts in the crowd, under the assumption that only experts have consistent evaluations. However, empirical results from a real human crowd indicate this assumption may not hold even on a simple engineering design evaluation task, as clusters of consistently wrong evaluators are shown to exist along with the cluster of experts. This suggests that both averaging evaluations and a crowd consensus model that relies only on evaluations may not be adequate for engineering design tasks, accordingly calling for further research into methods of finding experts within the crowd.

AB - Crowdsourced evaluation is a promising method of evaluating engineering design attributes that require human input. The challenge is to correctly estimate scores using a massive and diverse crowd, particularly when only a small subset of evaluators has the expertise to give correct evaluations. Since averaging evaluations across all evaluators will result in an inaccurate crowd evaluation, this paper benchmarks a crowd consensus model that aims to identify experts such that their evaluations may be given more weight. Simulation results indicate this crowd consensus model outperforms averaging when it correctly identifies experts in the crowd, under the assumption that only experts have consistent evaluations. However, empirical results from a real human crowd indicate this assumption may not hold even on a simple engineering design evaluation task, as clusters of consistently wrong evaluators are shown to exist along with the cluster of experts. This suggests that both averaging evaluations and a crowd consensus model that relies only on evaluations may not be adequate for engineering design tasks, accordingly calling for further research into methods of finding experts within the crowd.

KW - Crowd consensus

KW - Crowdsourcing

KW - Design evaluation

KW - Evaluator expertise

UR - http://www.scopus.com/inward/record.url?scp=84994000145&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84994000145&partnerID=8YFLogxK

U2 - 10.1115/1.4029065

DO - 10.1115/1.4029065

M3 - Article

VL - 137

JO - Journal of Mechanical Design - Transactions of the ASME

JF - Journal of Mechanical Design - Transactions of the ASME

SN - 1050-0472

IS - 3

M1 - 031101

ER -