Formative evaluation of the accuracy of a clinical decision support system for cervical cancer screening

Kavishwar Balwant Wagholikar Dr., Kathy L. MacLaughlin, Thomas M. Kastner, Petra M. Casey, Michael Henry, Robert Greenes, Hongfang Liu, Rajeev Chaudhry

Research output: Contribution to journalArticle

21 Scopus citations

Abstract

Objectives: We previously developed and reported on a prototype clinical decision support system (CDSS) for cervical cancer screening. However, the system is complex as it is based on multiple guidelines and freetext processing. Therefore, the system is susceptible to failures. This report describes a formative evaluation of the system, which is a necessary step to ensure deployment readiness of the system. Materials and methods: Care providers who are potential end-users of the CDSS were invited to provide their recommendations for a random set of patients that represented diverse decision scenarios. The recommendations of the care providers and those generated by the CDSS were compared. Mismatched recommendations were reviewed by two independent experts. Results: A total of 25 users participated in this study and provided recommendations for 175 cases. The CDSS had an accuracy of 87% and 12 types of CDSS errors were identified, which were mainly due to deficiencies in the system's guideline rules. When the deficiencies were rectified, the CDSS generated optimal recommendations for all failure cases, except one with incomplete documentation. Discussion and conclusions: The crowd-sourcing approach for construction of the reference set, coupled with the expert review of mismatched recommendations, facilitated an effective evaluation and enhancement of the system, by identifying decision scenarios that were missed by the system's developers. The described methodology will be useful for other researchers who seek rapidly to evaluate and enhance the deployment readiness of complex decision support systems.

Original languageEnglish (US)
Pages (from-to)749-757
Number of pages9
JournalJournal of the American Medical Informatics Association
Volume20
Issue number4
DOIs
StatePublished - 2013

ASJC Scopus subject areas

  • Health Informatics

Fingerprint Dive into the research topics of 'Formative evaluation of the accuracy of a clinical decision support system for cervical cancer screening'. Together they form a unique fingerprint.

  • Cite this