Analyzing Monte Carlo Simulation Studies With Classification and Regression Trees

Monte Carlo simulations are an important tool for researchers to study statistical properties of estimators, such as parameter bias, or the limits of various modeling approaches. Typically, the immense amount of data produced by Monte Carlo studies is analyzed with regression or analysis of variance, and researchers are faced with making arbitrary decisions regarding what effects to report and what interactions to test. Understanding current limitations, we propose a classification and regression trees (CART) approach from the statistical learning and data mining field to analyze Monte Carlo simulation data. We demonstrate the advantages of the CART approach and several extensions by reanalyzing and interpreting results from one published Monte Carlo study and one fully reproducible simulation example. Results suggest that CART is able to arrive at the same conclusions as current descriptive and inferential approaches and, at the same time, provide additional insight on the complex interactions among simulation factors.