Collaborative infrastructure for test-driven scientific model validation

One of the pillars of the modern scientific method is model validation: comparing a scientific model's predictions against empirical observations. Today, a scientist demonstrates the validity of a model by making an argument in a paper and submitting it for peer review, a process comparable to code review in software engineering. While human review helps to ensure that contributions meet high-level goals, software engineers typically supplement it with unit testing to get a more complete picture of the status of a project. We argue that a similar test-driven methodology would be valuable to scientific communities as they seek to validate increasingly complex models against growing repositories of empirical data. Scientific communities differ from software communities in several key ways, however. In this paper, we introduce SciUnit, a framework for test-driven scientific model validation, and outline how, supported by new and existing collaborative infrastructure, it could integrate into the modern scientific process.