Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR

Muhammad F. Walji, Elsbeth Kalenderian, Mark Piotrowski, Duong Tran, Krishna K. Kookal, Oluwabunmi Tokede, Joel M. White, Ram Vaderhobli, Rachel Ramoni, Paul C. Stark, Nicole S. Kimmes, Maxim Lagerweij, Vimla Patel

Research output: Contribution to journalArticle

27 Citations (Scopus)

Abstract

Objective: To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys. Materials and methods: Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N= 32), semi-structured interviews (N= 36), and surveys (N= 35). Results: The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods. Discussion: Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR. Conclusion: We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.

Original languageEnglish (US)
Pages (from-to)361-367
Number of pages7
JournalInternational Journal of Medical Informatics
Volume83
Issue number5
DOIs
StatePublished - Jan 1 2014
Externally publishedYes

Fingerprint

Interviews
Dental Faculties
Dental Students
Medical Informatics
Dental Schools
Tooth
Software
Surveys and Questionnaires
Therapeutics

Keywords

  • EHR
  • Human factors
  • Methodology
  • Usability

ASJC Scopus subject areas

  • Health Informatics

Cite this

Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR. / Walji, Muhammad F.; Kalenderian, Elsbeth; Piotrowski, Mark; Tran, Duong; Kookal, Krishna K.; Tokede, Oluwabunmi; White, Joel M.; Vaderhobli, Ram; Ramoni, Rachel; Stark, Paul C.; Kimmes, Nicole S.; Lagerweij, Maxim; Patel, Vimla.

In: International Journal of Medical Informatics, Vol. 83, No. 5, 01.01.2014, p. 361-367.

Research output: Contribution to journalArticle

Walji, MF, Kalenderian, E, Piotrowski, M, Tran, D, Kookal, KK, Tokede, O, White, JM, Vaderhobli, R, Ramoni, R, Stark, PC, Kimmes, NS, Lagerweij, M & Patel, V 2014, 'Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR', International Journal of Medical Informatics, vol. 83, no. 5, pp. 361-367. https://doi.org/10.1016/j.ijmedinf.2014.01.010
Walji, Muhammad F. ; Kalenderian, Elsbeth ; Piotrowski, Mark ; Tran, Duong ; Kookal, Krishna K. ; Tokede, Oluwabunmi ; White, Joel M. ; Vaderhobli, Ram ; Ramoni, Rachel ; Stark, Paul C. ; Kimmes, Nicole S. ; Lagerweij, Maxim ; Patel, Vimla. / Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR. In: International Journal of Medical Informatics. 2014 ; Vol. 83, No. 5. pp. 361-367.
@article{17220ce5f35841e6bd1b06b9d25d2e56,
title = "Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR",
abstract = "Objective: To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys. Materials and methods: Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N= 32), semi-structured interviews (N= 36), and surveys (N= 35). Results: The three methods together identified a total of 187 usability violations: 54{\%} via user testing, 28{\%} via the semi-structured interview and 18{\%} from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83{\%}), followed by the interview (63{\%}) and survey (29{\%}) methods. Discussion: Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR. Conclusion: We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.",
keywords = "EHR, Human factors, Methodology, Usability",
author = "Walji, {Muhammad F.} and Elsbeth Kalenderian and Mark Piotrowski and Duong Tran and Kookal, {Krishna K.} and Oluwabunmi Tokede and White, {Joel M.} and Ram Vaderhobli and Rachel Ramoni and Stark, {Paul C.} and Kimmes, {Nicole S.} and Maxim Lagerweij and Vimla Patel",
year = "2014",
month = "1",
day = "1",
doi = "10.1016/j.ijmedinf.2014.01.010",
language = "English (US)",
volume = "83",
pages = "361--367",
journal = "International Journal of Medical Informatics",
issn = "1386-5056",
publisher = "Elsevier Ireland Ltd",
number = "5",

}

TY - JOUR

T1 - Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR

AU - Walji, Muhammad F.

AU - Kalenderian, Elsbeth

AU - Piotrowski, Mark

AU - Tran, Duong

AU - Kookal, Krishna K.

AU - Tokede, Oluwabunmi

AU - White, Joel M.

AU - Vaderhobli, Ram

AU - Ramoni, Rachel

AU - Stark, Paul C.

AU - Kimmes, Nicole S.

AU - Lagerweij, Maxim

AU - Patel, Vimla

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Objective: To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys. Materials and methods: Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N= 32), semi-structured interviews (N= 36), and surveys (N= 35). Results: The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods. Discussion: Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR. Conclusion: We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.

AB - Objective: To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys. Materials and methods: Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N= 32), semi-structured interviews (N= 36), and surveys (N= 35). Results: The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods. Discussion: Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR. Conclusion: We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.

KW - EHR

KW - Human factors

KW - Methodology

KW - Usability

UR - http://www.scopus.com/inward/record.url?scp=84897389158&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84897389158&partnerID=8YFLogxK

U2 - 10.1016/j.ijmedinf.2014.01.010

DO - 10.1016/j.ijmedinf.2014.01.010

M3 - Article

C2 - 24583045

AN - SCOPUS:84897389158

VL - 83

SP - 361

EP - 367

JO - International Journal of Medical Informatics

JF - International Journal of Medical Informatics

SN - 1386-5056

IS - 5

ER -