Evaluation of structural equation modeling estimates of reliability for scales with ordered categorical items

Different methods are reviewed for estimating reliability based on a single administration of a test consisting of ordered categorical items. The nonlinear SEM reliability coefficient appears more flexible than other methods because it is designed to assess the reliability of summed scores for ordered categorical items through the modeling of the item relationships. However, this method assumes that the variables underlying the ordered categorical items are normally distributed. We evaluated nonlinear and linear SEM reliability coefficients using a Monte Carlo study by manipulating factor structure, distribution of the continuous items, number of ordered categories, distribution of the ordered categorical items, and sample size. The nonlinear SEM reliability coefficient performed as well as or better than the linear SEM reliability coefficient in most conditions and was relatively robust to modest violations of the normality assumption. The nonlinear SEM reliability coefficient was substantially biased when categorical item distributions were extremely skewed and the skewness differed markedly across items. Bias in these coefficients may be diagnosable in practice by assessing model fit.