Collaborative knowledge acquisition for the design of context-aware alert systems

Erel Joffe, Ofer Havakuk, Jorge R. Herskovic, Vimla Patel, Elmer Victor Bernstam

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Objective To present a framework for combining implicit knowledge acquisition from multiple experts with machine learning and to evaluate this framework in the context of anemia alerts. Materials and Methods Five internal medicine residents reviewed 18 anemia alerts, while 'talking aloud'. They identified features that were reviewed by two or more physicians to determine appropriate alert level, etiology and treatment recommendation. Based on these features, data were extracted from 100 randomlyselected anemia cases for a training set and an additional 82 cases for a test set. Two staff internists assigned an alert level, etiology and treatment recommendation before and after reviewing the entire electronic medical record. The training set of 118 cases (100 plus 18) and the test set of 82 cases were explored using RIDOR and JRip algorithms. Results The feature set was sufficient to assess 93% of anemia cases (intraclass correlation for alert level before and after review of the records by internists 1 and 2 were 0.92 and 0.95, respectively). High-precision classifiers were constructed to identify low-level alerts (precision p=0.87, recall R=0.4), iron deficiency (p=1.0, R=0.73), and anemia associated with kidney disease (p=0.87, R=0.77). Discussion It was possible to identify low-level alerts and several conditions commonly associated with chronic anemia. This approach may reduce the number of clinically unimportant alerts. The study was limited to anemia alerts. Furthermore, clinicians were aware of the study hypotheses potentially biasing their evaluation. Conclusion Implicit knowledge acquisition, collaborative filtering and machine learning were combined automatically to induce clinically meaningful and precise decision rules.

Original languageEnglish (US)
Pages (from-to)988-994
Number of pages7
JournalJournal of the American Medical Informatics Association
Volume19
Issue number6
DOIs
StatePublished - Nov 1 2012
Externally publishedYes

Fingerprint

Anemia
Electronic Health Records
Kidney Diseases
Internal Medicine
Iron
Physicians
Therapeutics

ASJC Scopus subject areas

  • Health Informatics

Cite this

Collaborative knowledge acquisition for the design of context-aware alert systems. / Joffe, Erel; Havakuk, Ofer; Herskovic, Jorge R.; Patel, Vimla; Bernstam, Elmer Victor.

In: Journal of the American Medical Informatics Association, Vol. 19, No. 6, 01.11.2012, p. 988-994.

Research output: Contribution to journalArticle

Joffe, Erel ; Havakuk, Ofer ; Herskovic, Jorge R. ; Patel, Vimla ; Bernstam, Elmer Victor. / Collaborative knowledge acquisition for the design of context-aware alert systems. In: Journal of the American Medical Informatics Association. 2012 ; Vol. 19, No. 6. pp. 988-994.
@article{f69440a053394222b3077395ac5dab0e,
title = "Collaborative knowledge acquisition for the design of context-aware alert systems",
abstract = "Objective To present a framework for combining implicit knowledge acquisition from multiple experts with machine learning and to evaluate this framework in the context of anemia alerts. Materials and Methods Five internal medicine residents reviewed 18 anemia alerts, while 'talking aloud'. They identified features that were reviewed by two or more physicians to determine appropriate alert level, etiology and treatment recommendation. Based on these features, data were extracted from 100 randomlyselected anemia cases for a training set and an additional 82 cases for a test set. Two staff internists assigned an alert level, etiology and treatment recommendation before and after reviewing the entire electronic medical record. The training set of 118 cases (100 plus 18) and the test set of 82 cases were explored using RIDOR and JRip algorithms. Results The feature set was sufficient to assess 93{\%} of anemia cases (intraclass correlation for alert level before and after review of the records by internists 1 and 2 were 0.92 and 0.95, respectively). High-precision classifiers were constructed to identify low-level alerts (precision p=0.87, recall R=0.4), iron deficiency (p=1.0, R=0.73), and anemia associated with kidney disease (p=0.87, R=0.77). Discussion It was possible to identify low-level alerts and several conditions commonly associated with chronic anemia. This approach may reduce the number of clinically unimportant alerts. The study was limited to anemia alerts. Furthermore, clinicians were aware of the study hypotheses potentially biasing their evaluation. Conclusion Implicit knowledge acquisition, collaborative filtering and machine learning were combined automatically to induce clinically meaningful and precise decision rules.",
author = "Erel Joffe and Ofer Havakuk and Herskovic, {Jorge R.} and Vimla Patel and Bernstam, {Elmer Victor}",
year = "2012",
month = "11",
day = "1",
doi = "10.1136/amiajnl-2012-000849",
language = "English (US)",
volume = "19",
pages = "988--994",
journal = "Journal of the American Medical Informatics Association : JAMIA",
issn = "1067-5027",
publisher = "Oxford University Press",
number = "6",

}

TY - JOUR

T1 - Collaborative knowledge acquisition for the design of context-aware alert systems

AU - Joffe, Erel

AU - Havakuk, Ofer

AU - Herskovic, Jorge R.

AU - Patel, Vimla

AU - Bernstam, Elmer Victor

PY - 2012/11/1

Y1 - 2012/11/1

N2 - Objective To present a framework for combining implicit knowledge acquisition from multiple experts with machine learning and to evaluate this framework in the context of anemia alerts. Materials and Methods Five internal medicine residents reviewed 18 anemia alerts, while 'talking aloud'. They identified features that were reviewed by two or more physicians to determine appropriate alert level, etiology and treatment recommendation. Based on these features, data were extracted from 100 randomlyselected anemia cases for a training set and an additional 82 cases for a test set. Two staff internists assigned an alert level, etiology and treatment recommendation before and after reviewing the entire electronic medical record. The training set of 118 cases (100 plus 18) and the test set of 82 cases were explored using RIDOR and JRip algorithms. Results The feature set was sufficient to assess 93% of anemia cases (intraclass correlation for alert level before and after review of the records by internists 1 and 2 were 0.92 and 0.95, respectively). High-precision classifiers were constructed to identify low-level alerts (precision p=0.87, recall R=0.4), iron deficiency (p=1.0, R=0.73), and anemia associated with kidney disease (p=0.87, R=0.77). Discussion It was possible to identify low-level alerts and several conditions commonly associated with chronic anemia. This approach may reduce the number of clinically unimportant alerts. The study was limited to anemia alerts. Furthermore, clinicians were aware of the study hypotheses potentially biasing their evaluation. Conclusion Implicit knowledge acquisition, collaborative filtering and machine learning were combined automatically to induce clinically meaningful and precise decision rules.

AB - Objective To present a framework for combining implicit knowledge acquisition from multiple experts with machine learning and to evaluate this framework in the context of anemia alerts. Materials and Methods Five internal medicine residents reviewed 18 anemia alerts, while 'talking aloud'. They identified features that were reviewed by two or more physicians to determine appropriate alert level, etiology and treatment recommendation. Based on these features, data were extracted from 100 randomlyselected anemia cases for a training set and an additional 82 cases for a test set. Two staff internists assigned an alert level, etiology and treatment recommendation before and after reviewing the entire electronic medical record. The training set of 118 cases (100 plus 18) and the test set of 82 cases were explored using RIDOR and JRip algorithms. Results The feature set was sufficient to assess 93% of anemia cases (intraclass correlation for alert level before and after review of the records by internists 1 and 2 were 0.92 and 0.95, respectively). High-precision classifiers were constructed to identify low-level alerts (precision p=0.87, recall R=0.4), iron deficiency (p=1.0, R=0.73), and anemia associated with kidney disease (p=0.87, R=0.77). Discussion It was possible to identify low-level alerts and several conditions commonly associated with chronic anemia. This approach may reduce the number of clinically unimportant alerts. The study was limited to anemia alerts. Furthermore, clinicians were aware of the study hypotheses potentially biasing their evaluation. Conclusion Implicit knowledge acquisition, collaborative filtering and machine learning were combined automatically to induce clinically meaningful and precise decision rules.

UR - http://www.scopus.com/inward/record.url?scp=84867665946&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84867665946&partnerID=8YFLogxK

U2 - 10.1136/amiajnl-2012-000849

DO - 10.1136/amiajnl-2012-000849

M3 - Article

VL - 19

SP - 988

EP - 994

JO - Journal of the American Medical Informatics Association : JAMIA

JF - Journal of the American Medical Informatics Association : JAMIA

SN - 1067-5027

IS - 6

ER -