Reliability in Coding Open-Ended Data

Lessons Learned from HIV Behavioral Research

Daniel Hruschka, Deborah Schwartz, Daphne Cobb St.john, Erin Picone-Decaro, Richard A. Jenkins, James W. Carey

Research output: Contribution to journalArticle

307 Citations (Scopus)

Abstract

Analysis of text from open-ended interviews has become an important research tool in numerous fields, including business, education, and health research. Coding is an essential part of such analysis, but questions of quality control in the coding process have generally received little attention. This article examines the text coding process applied to three HIV-related studies conducted with the Centers for Disease Control and Prevention considering populations in the United States and Zimbabwe. Based on experience coding data from these studies, we conclude that (1) a team of coders will initially produce very different codings, but (2) it is possible, through a process of codebook revision and recoding, to establish strong levels of intercoder reliability (e.g., most codes with kappa 0.8). Furthermore, steps can be taken to improve initially poor intercoder reliability and to reduce the number of iterations required to generate stronger intercoder reliability.

Original languageEnglish (US)
Pages (from-to)307-331
Number of pages25
JournalField Methods
Volume16
Issue number3
DOIs
StatePublished - Jan 1 2004
Externally publishedYes

Fingerprint

behavioral research
coding
business education
quality control
Zimbabwe
Disease
interview
health
experience

Keywords

  • data
  • intercoder agreement
  • interrater agreement
  • open-ended
  • qualitative
  • reliability

ASJC Scopus subject areas

  • Anthropology

Cite this

Hruschka, D., Schwartz, D., St.john, D. C., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004). Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research. Field Methods, 16(3), 307-331. https://doi.org/10.1177/1525822X04266540

Reliability in Coding Open-Ended Data : Lessons Learned from HIV Behavioral Research. / Hruschka, Daniel; Schwartz, Deborah; St.john, Daphne Cobb; Picone-Decaro, Erin; Jenkins, Richard A.; Carey, James W.

In: Field Methods, Vol. 16, No. 3, 01.01.2004, p. 307-331.

Research output: Contribution to journalArticle

Hruschka, D, Schwartz, D, St.john, DC, Picone-Decaro, E, Jenkins, RA & Carey, JW 2004, 'Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research', Field Methods, vol. 16, no. 3, pp. 307-331. https://doi.org/10.1177/1525822X04266540
Hruschka, Daniel ; Schwartz, Deborah ; St.john, Daphne Cobb ; Picone-Decaro, Erin ; Jenkins, Richard A. ; Carey, James W. / Reliability in Coding Open-Ended Data : Lessons Learned from HIV Behavioral Research. In: Field Methods. 2004 ; Vol. 16, No. 3. pp. 307-331.
@article{8119a8d0f72944969536c950ee3806e1,
title = "Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research",
abstract = "Analysis of text from open-ended interviews has become an important research tool in numerous fields, including business, education, and health research. Coding is an essential part of such analysis, but questions of quality control in the coding process have generally received little attention. This article examines the text coding process applied to three HIV-related studies conducted with the Centers for Disease Control and Prevention considering populations in the United States and Zimbabwe. Based on experience coding data from these studies, we conclude that (1) a team of coders will initially produce very different codings, but (2) it is possible, through a process of codebook revision and recoding, to establish strong levels of intercoder reliability (e.g., most codes with kappa 0.8). Furthermore, steps can be taken to improve initially poor intercoder reliability and to reduce the number of iterations required to generate stronger intercoder reliability.",
keywords = "data, intercoder agreement, interrater agreement, open-ended, qualitative, reliability",
author = "Daniel Hruschka and Deborah Schwartz and St.john, {Daphne Cobb} and Erin Picone-Decaro and Jenkins, {Richard A.} and Carey, {James W.}",
year = "2004",
month = "1",
day = "1",
doi = "10.1177/1525822X04266540",
language = "English (US)",
volume = "16",
pages = "307--331",
journal = "Field Methods",
issn = "1525-822X",
publisher = "SAGE Publications Inc.",
number = "3",

}

TY - JOUR

T1 - Reliability in Coding Open-Ended Data

T2 - Lessons Learned from HIV Behavioral Research

AU - Hruschka, Daniel

AU - Schwartz, Deborah

AU - St.john, Daphne Cobb

AU - Picone-Decaro, Erin

AU - Jenkins, Richard A.

AU - Carey, James W.

PY - 2004/1/1

Y1 - 2004/1/1

N2 - Analysis of text from open-ended interviews has become an important research tool in numerous fields, including business, education, and health research. Coding is an essential part of such analysis, but questions of quality control in the coding process have generally received little attention. This article examines the text coding process applied to three HIV-related studies conducted with the Centers for Disease Control and Prevention considering populations in the United States and Zimbabwe. Based on experience coding data from these studies, we conclude that (1) a team of coders will initially produce very different codings, but (2) it is possible, through a process of codebook revision and recoding, to establish strong levels of intercoder reliability (e.g., most codes with kappa 0.8). Furthermore, steps can be taken to improve initially poor intercoder reliability and to reduce the number of iterations required to generate stronger intercoder reliability.

AB - Analysis of text from open-ended interviews has become an important research tool in numerous fields, including business, education, and health research. Coding is an essential part of such analysis, but questions of quality control in the coding process have generally received little attention. This article examines the text coding process applied to three HIV-related studies conducted with the Centers for Disease Control and Prevention considering populations in the United States and Zimbabwe. Based on experience coding data from these studies, we conclude that (1) a team of coders will initially produce very different codings, but (2) it is possible, through a process of codebook revision and recoding, to establish strong levels of intercoder reliability (e.g., most codes with kappa 0.8). Furthermore, steps can be taken to improve initially poor intercoder reliability and to reduce the number of iterations required to generate stronger intercoder reliability.

KW - data

KW - intercoder agreement

KW - interrater agreement

KW - open-ended

KW - qualitative

KW - reliability

UR - http://www.scopus.com/inward/record.url?scp=33444476650&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33444476650&partnerID=8YFLogxK

U2 - 10.1177/1525822X04266540

DO - 10.1177/1525822X04266540

M3 - Article

VL - 16

SP - 307

EP - 331

JO - Field Methods

JF - Field Methods

SN - 1525-822X

IS - 3

ER -