Efficacy of peer review network structures

The effects of reciprocity and clustering

Scott Stevens, Andrew Waters, Dmytro Babik, David Tinapple

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

IT-enabled peer-based creation, review, and evaluation systems are widely spread in multiple areas of open innovation and knowledge management. Despite a noticeable variety of designs, particularly, in the structure of peer networks (that is, how participants are linked to each other as creators and reviewers), these design choices are hardly ever grounded in design research. Characteristics of peer network structure, such as reciprocity and clustering, may affect how well such systems reveal participants' competencies and their products' qualities. Designing peer review systems that produce valid and reliable evaluations is, therefore, among the most fundamental concerns. Using a simulation approach, we show that reciprocity and clustering indeed have an effect, but its direction and magnitude depend on the evaluation scale used. So far, we have found no evidence that transitional networks have superior efficacy in comparison with "pure" networks. We outlined directions for further investigation.

Original languageEnglish (US)
Title of host publication2016 International Conference on Information Systems, ICIS 2016
PublisherAssociation for Information Systems
ISBN (Electronic)9780996683135
StatePublished - 2016
Event2016 International Conference on Information Systems, ICIS 2016 - Dublin, Ireland
Duration: Dec 11 2016Dec 14 2016

Other

Other2016 International Conference on Information Systems, ICIS 2016
CountryIreland
CityDublin
Period12/11/1612/14/16

Fingerprint

Knowledge management
Innovation

Keywords

  • Clustering
  • Evaluation systems
  • Information quality
  • Knowledge artifacts
  • Network structures
  • Peer review
  • Reciprocity

ASJC Scopus subject areas

  • Information Systems

Cite this

Stevens, S., Waters, A., Babik, D., & Tinapple, D. (2016). Efficacy of peer review network structures: The effects of reciprocity and clustering. In 2016 International Conference on Information Systems, ICIS 2016 Association for Information Systems.

Efficacy of peer review network structures : The effects of reciprocity and clustering. / Stevens, Scott; Waters, Andrew; Babik, Dmytro; Tinapple, David.

2016 International Conference on Information Systems, ICIS 2016. Association for Information Systems, 2016.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Stevens, S, Waters, A, Babik, D & Tinapple, D 2016, Efficacy of peer review network structures: The effects of reciprocity and clustering. in 2016 International Conference on Information Systems, ICIS 2016. Association for Information Systems, 2016 International Conference on Information Systems, ICIS 2016, Dublin, Ireland, 12/11/16.
Stevens S, Waters A, Babik D, Tinapple D. Efficacy of peer review network structures: The effects of reciprocity and clustering. In 2016 International Conference on Information Systems, ICIS 2016. Association for Information Systems. 2016
Stevens, Scott ; Waters, Andrew ; Babik, Dmytro ; Tinapple, David. / Efficacy of peer review network structures : The effects of reciprocity and clustering. 2016 International Conference on Information Systems, ICIS 2016. Association for Information Systems, 2016.
@inproceedings{e14624bede234ea29cc95f71f3ecae6f,
title = "Efficacy of peer review network structures: The effects of reciprocity and clustering",
abstract = "IT-enabled peer-based creation, review, and evaluation systems are widely spread in multiple areas of open innovation and knowledge management. Despite a noticeable variety of designs, particularly, in the structure of peer networks (that is, how participants are linked to each other as creators and reviewers), these design choices are hardly ever grounded in design research. Characteristics of peer network structure, such as reciprocity and clustering, may affect how well such systems reveal participants' competencies and their products' qualities. Designing peer review systems that produce valid and reliable evaluations is, therefore, among the most fundamental concerns. Using a simulation approach, we show that reciprocity and clustering indeed have an effect, but its direction and magnitude depend on the evaluation scale used. So far, we have found no evidence that transitional networks have superior efficacy in comparison with {"}pure{"} networks. We outlined directions for further investigation.",
keywords = "Clustering, Evaluation systems, Information quality, Knowledge artifacts, Network structures, Peer review, Reciprocity",
author = "Scott Stevens and Andrew Waters and Dmytro Babik and David Tinapple",
year = "2016",
language = "English (US)",
booktitle = "2016 International Conference on Information Systems, ICIS 2016",
publisher = "Association for Information Systems",

}

TY - GEN

T1 - Efficacy of peer review network structures

T2 - The effects of reciprocity and clustering

AU - Stevens, Scott

AU - Waters, Andrew

AU - Babik, Dmytro

AU - Tinapple, David

PY - 2016

Y1 - 2016

N2 - IT-enabled peer-based creation, review, and evaluation systems are widely spread in multiple areas of open innovation and knowledge management. Despite a noticeable variety of designs, particularly, in the structure of peer networks (that is, how participants are linked to each other as creators and reviewers), these design choices are hardly ever grounded in design research. Characteristics of peer network structure, such as reciprocity and clustering, may affect how well such systems reveal participants' competencies and their products' qualities. Designing peer review systems that produce valid and reliable evaluations is, therefore, among the most fundamental concerns. Using a simulation approach, we show that reciprocity and clustering indeed have an effect, but its direction and magnitude depend on the evaluation scale used. So far, we have found no evidence that transitional networks have superior efficacy in comparison with "pure" networks. We outlined directions for further investigation.

AB - IT-enabled peer-based creation, review, and evaluation systems are widely spread in multiple areas of open innovation and knowledge management. Despite a noticeable variety of designs, particularly, in the structure of peer networks (that is, how participants are linked to each other as creators and reviewers), these design choices are hardly ever grounded in design research. Characteristics of peer network structure, such as reciprocity and clustering, may affect how well such systems reveal participants' competencies and their products' qualities. Designing peer review systems that produce valid and reliable evaluations is, therefore, among the most fundamental concerns. Using a simulation approach, we show that reciprocity and clustering indeed have an effect, but its direction and magnitude depend on the evaluation scale used. So far, we have found no evidence that transitional networks have superior efficacy in comparison with "pure" networks. We outlined directions for further investigation.

KW - Clustering

KW - Evaluation systems

KW - Information quality

KW - Knowledge artifacts

KW - Network structures

KW - Peer review

KW - Reciprocity

UR - http://www.scopus.com/inward/record.url?scp=85019501155&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85019501155&partnerID=8YFLogxK

M3 - Conference contribution

BT - 2016 International Conference on Information Systems, ICIS 2016

PB - Association for Information Systems

ER -