Digital medicine and the curse of dimensionality

Visar Berisha, Chelsea Krantsevich, P. Richard Hahn, Shira Hahn, Gautam Dasarathy, Pavan Turaga, Julie Liss

Research output: Contribution to journalReview articlepeer-review

Abstract

Digital health data are multimodal and high-dimensional. A patient’s health state can be characterized by a multitude of signals including medical imaging, clinical variables, genome sequencing, conversations between clinicians and patients, and continuous signals from wearables, among others. This high volume, personalized data stream aggregated over patients’ lives has spurred interest in developing new artificial intelligence (AI) models for higher-precision diagnosis, prognosis, and tracking. While the promise of these algorithms is undeniable, their dissemination and adoption have been slow, owing partially to unpredictable AI model performance once deployed in the real world. We posit that one of the rate-limiting factors in developing algorithms that generalize to real-world scenarios is the very attribute that makes the data exciting—their high-dimensional nature. This paper considers how the large number of features in vast digital health data can challenge the development of robust AI models—a phenomenon known as “the curse of dimensionality” in statistical learning theory. We provide an overview of the curse of dimensionality in the context of digital health, demonstrate how it can negatively impact out-of-sample performance, and highlight important considerations for researchers and algorithm designers.

Original languageEnglish (US)
Article number153
Journalnpj Digital Medicine
Volume4
Issue number1
DOIs
StatePublished - Dec 2021

ASJC Scopus subject areas

  • Medicine (miscellaneous)
  • Computer Science Applications
  • Health Informatics
  • Health Information Management

Fingerprint

Dive into the research topics of 'Digital medicine and the curse of dimensionality'. Together they form a unique fingerprint.

Cite this