TY - JOUR
T1 - Coding Qualitative Data at Scale
T2 - Guidance for Large Coder Teams Based on 18 Studies
AU - Beresford, Melissa
AU - Wutich, Amber
AU - du Bray, Margaret V.
AU - Ruth, Alissa
AU - Stotts, Rhian
AU - SturtzSreetharan, Cindi
AU - Brewis, Alexandra
N1 - Funding Information:
We acknowledge funding which supported the projects we describe, provided by U.S. National Science Foundation (Awards BCS-2017491, BCS-1759972, GCR-2021147, SES-1462086) and the Virginia G. Piper Charitable Trust (to Mayo Clinic-ASU Obesity Solutions initiative).
Funding Information:
We gratefully acknowledge CHELab managers Meredith Gartin, Christopher Roberts, Charlayne Mitchell, and Mirtha Garcia Reyes and CHEL-affiliated postdoctoral scholars Roseanne Schuster, Julia (Chrissie) Bausch, and Ana?s Roque. We also thank the students and colleagues who collaborated on CHEL research over the last 15 years. Many provided valuable feedback as we developed the procedures described here. We acknowledge funding which supported the projects we describe, provided by U.S. National Science Foundation (Awards BCS-2017491, BCS-1759972, GCR-2021147, SES-1462086) and the Virginia G. Piper Charitable Trust (to Mayo Clinic-ASU Obesity Solutions initiative).
Publisher Copyright:
© The Author(s) 2022.
PY - 2022/1/11
Y1 - 2022/1/11
N2 - We outline a process for using large coder teams (10 + coders) to code large-scale qualitative data sets. The process reflects experience recruiting and managing large teams of novice and trainee coders for 18 projects in the last decade, each engaging a coding team of 12 (minimum) to 54 (maximum) coders. We identify four unique challenges to large coder teams that are not presently discussed in the methodological literature: (1) recruiting and training coders, (2) providing coder compensation and incentives, (3) maintaining data quality and ensuring coding reliability at scale, and (4) building team cohesion and morale. For each challenge, we provide associated guidance. We conclude with a discussion of advantages and disadvantages of large coder teams for qualitative research and provide notes of caution for anyone considering hiring and/or managing large coder teams for research (whether in academia, government and non-profit sectors, or industry).
AB - We outline a process for using large coder teams (10 + coders) to code large-scale qualitative data sets. The process reflects experience recruiting and managing large teams of novice and trainee coders for 18 projects in the last decade, each engaging a coding team of 12 (minimum) to 54 (maximum) coders. We identify four unique challenges to large coder teams that are not presently discussed in the methodological literature: (1) recruiting and training coders, (2) providing coder compensation and incentives, (3) maintaining data quality and ensuring coding reliability at scale, and (4) building team cohesion and morale. For each challenge, we provide associated guidance. We conclude with a discussion of advantages and disadvantages of large coder teams for qualitative research and provide notes of caution for anyone considering hiring and/or managing large coder teams for research (whether in academia, government and non-profit sectors, or industry).
KW - collaborative research
KW - intercoder agreement
KW - intercoder reliability
KW - qualitative coding
KW - qualitative data analysis
KW - team-based coding
KW - text analysis
UR - http://www.scopus.com/inward/record.url?scp=85128232900&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85128232900&partnerID=8YFLogxK
U2 - 10.1177/16094069221075860
DO - 10.1177/16094069221075860
M3 - Article
AN - SCOPUS:85128232900
SN - 1609-4069
VL - 21
JO - International Journal of Qualitative Methods
JF - International Journal of Qualitative Methods
ER -