TY - JOUR
T1 - The Performance of Multivariate Methods for Two-Group Comparisons with Small Samples and Incomplete Data
AU - Pituch, Keenan A.
AU - Joshi, Megha
AU - Cain, Molly E.
AU - Whittaker, Tiffany A.
AU - Chang, Wanchen
AU - Park, Ryoungsun
AU - McDougall, Graham J.
N1 - Funding Information:
This work was supported by Grant R01 AG 15384 from the National Institutes of Health, National Institute on Aging.
PY - 2020/9/2
Y1 - 2020/9/2
N2 - In intervention studies having multiple outcomes, researchers often use a series of univariate tests (e.g., ANOVAs) to assess group mean differences. Previous research found that this approach properly controls Type I error and generally provides greater power compared to MANOVA, especially under realistic effect size and correlation combinations. However, when group differences are assessed for a specific outcome, these procedures are strictly univariate and do not consider the outcome correlations, which may be problematic with missing outcome data. Linear mixed or multivariate multilevel models (MVMMs), implemented with maximum likelihood estimation, present an alternative analysis option where outcome correlations are taken into account when specific group mean differences are estimated. In this study, we use simulation methods to compare the performance of separate independent samples t tests estimated with ordinary least squares and analogous t tests from MVMMs to assess two-group mean differences with multiple outcomes under small sample and missingness conditions. Study results indicated that a MVMM implemented with restricted maximum likelihood estimation combined with the Kenward–Roger correction had the best performance. Therefore, for intervention studies with small N and normally distributed multivariate outcomes, the Kenward–Roger procedure is recommended over traditional methods and conventional MVMM analyses, particularly with incomplete data.
AB - In intervention studies having multiple outcomes, researchers often use a series of univariate tests (e.g., ANOVAs) to assess group mean differences. Previous research found that this approach properly controls Type I error and generally provides greater power compared to MANOVA, especially under realistic effect size and correlation combinations. However, when group differences are assessed for a specific outcome, these procedures are strictly univariate and do not consider the outcome correlations, which may be problematic with missing outcome data. Linear mixed or multivariate multilevel models (MVMMs), implemented with maximum likelihood estimation, present an alternative analysis option where outcome correlations are taken into account when specific group mean differences are estimated. In this study, we use simulation methods to compare the performance of separate independent samples t tests estimated with ordinary least squares and analogous t tests from MVMMs to assess two-group mean differences with multiple outcomes under small sample and missingness conditions. Study results indicated that a MVMM implemented with restricted maximum likelihood estimation combined with the Kenward–Roger correction had the best performance. Therefore, for intervention studies with small N and normally distributed multivariate outcomes, the Kenward–Roger procedure is recommended over traditional methods and conventional MVMM analyses, particularly with incomplete data.
KW - ANOVA
KW - Kenward–Roger correction
KW - missing data
KW - multivariate multilevel model
KW - small samples
UR - http://www.scopus.com/inward/record.url?scp=85073952633&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85073952633&partnerID=8YFLogxK
U2 - 10.1080/00273171.2019.1667217
DO - 10.1080/00273171.2019.1667217
M3 - Article
C2 - 31552754
AN - SCOPUS:85073952633
VL - 55
SP - 704
EP - 721
JO - Multivariate Behavioral Research
JF - Multivariate Behavioral Research
SN - 0027-3171
IS - 5
ER -