Interrater reliability of grading strength of evidence varies with the complexity of the evidence in systematic reviews

Nancy D. Berkman, Kathleen N. Lohr, Laura C. Morgan, Tzy Mey Kuo, Sally C. Morton

Research output: Contribution to journalArticlepeer-review

11 Scopus citations

Abstract

Objectives To examine consistency (interrater reliability) of applying guidance for grading strength of evidence in systematic reviews for the Agency for Healthcare Research and Quality Evidence-based Practice Center program. Study Design and Setting Using data from two systematic reviews, authors tested the main components of the approach: (1) scoring evidence on the four required domains (risk of bias, consistency, directness, and precision) separately for randomized controlled trials (RCTs) and observational studies and (2) developing an overall strength of evidence grade, given the scores for each of these domains. Results Conclusions about overall strength of evidence reached by experienced systematic reviewers based on the same evidence can differ greatly, especially for complex bodies of evidence. Current instructions may be sufficient for straightforward quantitative evaluations that use meta-analysis for summarizing RCT findings. In contrast, agreement suffered when evaluations did not lend themselves to meta-analysis and reviewers needed to rely on their own qualitative judgment. Three areas raised particular concern: (1) evidence from a combination of RCTs and observational studies, (2) outcomes with differing measurement, and (3) evidence that appeared to show no differences in outcomes. Conclusion Interrater reliability was highly variable for scoring strength of evidence domains and combining scores to reach overall strength of evidence grades. Future research can help in establishing improved methods for evaluating these complex bodies of evidence.

Original languageEnglish (US)
Pages (from-to)1105-1117.e1
JournalJournal of Clinical Epidemiology
Volume66
Issue number10
DOIs
StatePublished - Oct 2013
Externally publishedYes

Keywords

  • Agency for Healthcare Research and Quality
  • Comparative effectiveness
  • Evidence-based practice
  • Interrater reliability
  • Strength of evidence
  • Systematic review methodology

ASJC Scopus subject areas

  • Epidemiology

Fingerprint Dive into the research topics of 'Interrater reliability of grading strength of evidence varies with the complexity of the evidence in systematic reviews'. Together they form a unique fingerprint.

Cite this