Analyzing Longitudinal Multirater Data with Changing and Stable Raters

One issue in analyzing longitudinal multirater data arises if raters drop-in or drop-out throughout a longitudinal study. We term this issue random rater movement (RRM), assuming that the selection of raters into the study approximates a random process and is strongly ignorable. We explain how RRM can be modeled in case of longitudinal multirater designs with (a) interchangeable raters or (b) structurally different raters. To analyze measurement designs with stable and changing interchangeable raters, we recommend using a longitudinal multilevel confirmatory factor model. To analyze measurement designs with stable and changing structurally different raters, we propose a longitudinal multigroup confirmatory factor model. The proposed model is illustrated using real data. Additionally, the performance of the models with regard to a small number of raters and a relatively small overall sample size is examined in Monte Carlo simulation studies. Future directions for analyzing rater movement over time are provided.