The Effects of Dispersion and Reciprocity on Assessment Fidelity in Peer-Review Systems: A Simulation Study

Dmytro Babik, Scott P. Stevens, Andrew Waters, David Tinapple

Research output: Contribution to journalArticle

Abstract

Over the last 20 years, online peer review and assessment have become widely used and well-researched practices in education. Their use increased, especially with the proliferation of non-conventional large-scale and online modes of teaching and learning, such as MOOCs. A well-designed peer-review system is expected to produce valid and reliable assessments of the artifacts created by participants. These systems notably vary in designs, particularly in the structure of the peer-review networks; i.e., how participants are linked to each other as creators and reviewers. To date, little research has been done on how different network structures impact a system's ability to accurately assess the quality of the evaluated artifacts. We begin to address this gap, using a simulation approach to demonstrate that two network characteristics-dispersion and reciprocity-may indeed affect assessment fidelity in peer-review systems in conjunction with other design choices, such as evaluation scale and aggregation method. We also outline directions for further investigations of peer-review systems design.

Original languageEnglish (US)
JournalIEEE Transactions on Learning Technologies
DOIs
StateAccepted/In press - Jan 1 2020

Keywords

  • Assessment
  • clustering
  • dispersion
  • evaluation systems
  • information quality
  • knowledge artifacts
  • network structures
  • peer assessment
  • peer review
  • reciprocity

ASJC Scopus subject areas

  • Education
  • Engineering(all)
  • Computer Science Applications

Fingerprint Dive into the research topics of 'The Effects of Dispersion and Reciprocity on Assessment Fidelity in Peer-Review Systems: A Simulation Study'. Together they form a unique fingerprint.

  • Cite this