TY - JOUR
T1 - Generalizability of heterogeneous treatment effects based on causal forests applied to two randomized clinical trials of intensive glycemic control
AU - Raghavan, Sridharan
AU - Josey, Kevin
AU - Bahn, Gideon
AU - Reda, Domenic
AU - Basu, Sanjay
AU - Berkowitz, Seth A.
AU - Emanuele, Nicholas
AU - Reaven, Peter
AU - Ghosh, Debashis
N1 - Funding Information:
Declaration of interests: The authors declare the following financial interests and/or personal relationships which may be considered as potential competing interests: Sridharan Raghavan reports financial support was provided by American Heart Association. Debashis Ghosh reports financial support was provided by National Science Foundation. Medications and financial support for the VADT Study were provided by Sanofi, GlaxoSmithKline, Novo Nordisk, Roche, Kos Pharmaceuticals, Merck, and Amylin. This manuscript describes a secondary analysis of data from the VADT study but did not receive any financial or material support from the above companies.
Funding Information:
The authors have no conflicts of interest relevant to the work in this study. Medications and financial support were provided by Sanofi, GlaxoSmithKline, Novo Nordisk, Roche, Kos Pharmaceuticals, Merck, and Amylin. No other potential conflicts of interest relevant to this article were reported. These companies had no role in the design of the study, in the accrual or analysis of the data, or in the preparation or approval of the manuscript.
Funding Information:
This work was supported by the US Department of Veterans Affairs ( Award IK2-CX001907 to SR, Clinical Sciences Research & Development Service's Cooperative Studies Program #465-F to GB, and Cooperative Studies Program #2008 to NE); the American Heart Association ( Award 17MCPRP33670728 to SR); the National Science Foundation ( Award DMS 1914937 to DG ); the National Cancer Institute of the US National Institutes of Health ( Award R01 CA129102 to DG); and the National Institute of Diabetes and Digestive and Kidney Diseases of the US National Institutes of Health ( Award K23DK109200 to SAB). The VADT Study was supported by the Veterans Affairs Cooperative Studies Program, Department of Veterans Affairs Office of Research and Development; the American Diabetes Association; and the National Eye Institute. Medications and financial support for the VADT Study were provided by Sanofi, GlaxoSmithKline, Novo Nordisk, Roche, Kos Pharmaceuticals, Merck, and Amylin.
Publisher Copyright:
© 2021
PY - 2022/1
Y1 - 2022/1
N2 - Purpose Machine learning is an attractive tool for identifying heterogeneous treatment effects (HTE) of interventions but generalizability of machine learning derived HTE remains unclear. We examined generalizability of HTE detected using causal forests in two similarly designed randomized trials in type II diabetes patients. Methods We evaluated published HTE of intensive versus standard glycemic control on all-cause mortality from the Action to Control Cardiovascular Risk in Diabetes study (ACCORD) in a second trial, the Veterans Affairs Diabetes Trial (VADT). We then applied causal forests to VADT, ACCORD, and pooled data from both studies and compared variable importance and subgroup effects across samples. Results HTE in ACCORD did not replicate in similar subgroups in VADT, but variable importance was correlated between VADT and ACCORD (Kendall's tau-b 0.75). Applying causal forests to pooled individual-level data yielded seven subgroups with similar HTE across both studies, ranging from risk difference of all-cause mortality of -3.9% (95% CI -7.0, -0.8) to 4.7% (95% CI 1.8, 7.5). Conclusions Machine learning detection of HTE subgroups from randomized trials may not generalize across study samples even when variable importance is correlated. Pooling individual-level data may overcome differences in study populations and/or differences in interventions that limit HTE generalizability.
AB - Purpose Machine learning is an attractive tool for identifying heterogeneous treatment effects (HTE) of interventions but generalizability of machine learning derived HTE remains unclear. We examined generalizability of HTE detected using causal forests in two similarly designed randomized trials in type II diabetes patients. Methods We evaluated published HTE of intensive versus standard glycemic control on all-cause mortality from the Action to Control Cardiovascular Risk in Diabetes study (ACCORD) in a second trial, the Veterans Affairs Diabetes Trial (VADT). We then applied causal forests to VADT, ACCORD, and pooled data from both studies and compared variable importance and subgroup effects across samples. Results HTE in ACCORD did not replicate in similar subgroups in VADT, but variable importance was correlated between VADT and ACCORD (Kendall's tau-b 0.75). Applying causal forests to pooled individual-level data yielded seven subgroups with similar HTE across both studies, ranging from risk difference of all-cause mortality of -3.9% (95% CI -7.0, -0.8) to 4.7% (95% CI 1.8, 7.5). Conclusions Machine learning detection of HTE subgroups from randomized trials may not generalize across study samples even when variable importance is correlated. Pooling individual-level data may overcome differences in study populations and/or differences in interventions that limit HTE generalizability.
KW - BMI, Body mass index
KW - Generalizability, Glycemic control, Causal forests, Heterogeneous treatment effects. Abbreviations: ACCORD, Action to Control Cardiovascular Risk in Diabetes Study
KW - HGI, Hemoglobin glycation index
KW - HTE, Heterogeneous treatment effects
KW - HbA1c, Hemoglobin A1c
KW - VADT, Veterans Affairs Diabetes Trial
KW - eGFR, Estimated glomerular filtration rate
UR - http://www.scopus.com/inward/record.url?scp=85112093462&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85112093462&partnerID=8YFLogxK
U2 - 10.1016/j.annepidem.2021.07.003
DO - 10.1016/j.annepidem.2021.07.003
M3 - Article
C2 - 34280545
AN - SCOPUS:85112093462
VL - 65
SP - 101
EP - 108
JO - Annals of Epidemiology
JF - Annals of Epidemiology
SN - 1047-2797
ER -