TY - JOUR
T1 - Using data to enhance the expert panel process
T2 - Rating indications of alcohol-related problems in older adults
AU - Oishi, Sabine M.
AU - Morton, Sally C.
AU - Moore, Alison A.
AU - Beck, John C.
AU - Hays, Ron D.
AU - Spritzer, Karen L.
AU - Partridge, Jennifer M.
AU - Fink, Arlene
PY - 2001
Y1 - 2001
N2 - Objective: To enhance the validity of a well-known expert panel process, we used data from patient surveys to identify and correct rating errors. Methods: We used the two-round RAND/UCLA panel method to rate indications of harmful (presence of problems), hazardous (at risk for problems), and nonhazardous (no known risks) drinking in older adults. Results from the panel provided guidelines for classifying older individuals as harmful, hazardous, or nonhazardous drinkers, using a survey. The classifications yielded unexpectedly high numbers of harmful and hazardous drinkers. We hypothesized possible misclassifications of drinking risks and used the survey data to identify indications that may have led to invalid ratings. We modified problematic indications and asked three clinician panelists to evaluate the clinical usefulness of the modifications in a third panel round. We revised the indications based on panelist response and reexamined drinking classifications. Results: Using the original indications, 48% of drinkers in the sample were classified as harmful, 31% as hazardous, and 21% as nonhazardous. A review of the indications revealed framing bias in the original rating task and vague definitions of certain symptoms and conditions. The modified indications resulted in classifications of 22% harmful, 47% hazardous, and 31% nonhazardous drinkers. Conclusions: Analysis of survey data led to identification and correction of specific errors occurring during the panel-rating process. The validity of the RAND/UCLA method can be enhanced using data-driven modifications.
AB - Objective: To enhance the validity of a well-known expert panel process, we used data from patient surveys to identify and correct rating errors. Methods: We used the two-round RAND/UCLA panel method to rate indications of harmful (presence of problems), hazardous (at risk for problems), and nonhazardous (no known risks) drinking in older adults. Results from the panel provided guidelines for classifying older individuals as harmful, hazardous, or nonhazardous drinkers, using a survey. The classifications yielded unexpectedly high numbers of harmful and hazardous drinkers. We hypothesized possible misclassifications of drinking risks and used the survey data to identify indications that may have led to invalid ratings. We modified problematic indications and asked three clinician panelists to evaluate the clinical usefulness of the modifications in a third panel round. We revised the indications based on panelist response and reexamined drinking classifications. Results: Using the original indications, 48% of drinkers in the sample were classified as harmful, 31% as hazardous, and 21% as nonhazardous. A review of the indications revealed framing bias in the original rating task and vague definitions of certain symptoms and conditions. The modified indications resulted in classifications of 22% harmful, 47% hazardous, and 31% nonhazardous drinkers. Conclusions: Analysis of survey data led to identification and correction of specific errors occurring during the panel-rating process. The validity of the RAND/UCLA method can be enhanced using data-driven modifications.
KW - Aged
KW - Alcohol drinking
KW - Geriatric medicine
KW - Quality of health care
UR - http://www.scopus.com/inward/record.url?scp=0035027349&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=0035027349&partnerID=8YFLogxK
U2 - 10.1017/S0266462301104113
DO - 10.1017/S0266462301104113
M3 - Article
C2 - 11329839
AN - SCOPUS:0035027349
SN - 0266-4623
VL - 17
SP - 125
EP - 136
JO - International Journal of Technology Assessment in Health Care
JF - International Journal of Technology Assessment in Health Care
IS - 1
ER -