Evaluating temporal planning domains

The last eight years have seen dramatic progress in temporal planning as highlighted by the temporal track in the last three International Planning Competitions (IPC). However, our recent work, (Cushing et al. 2007), showed that most of the competition winning planners are only complete for very restricted forms of temporal planning languages that are in a sense indistinguishable from STRIPS. In this paper we consider the impact of those results on the design of benchmark temporal planning domains, and by extension, the temporal planning competition. We start by setting out to verify our speculation that the competition domains are temporally simple. This turns out to be tricky, and we develop a set of increasingly powerful analytic methods for domain analysis. Our analysis establishes that the benchmark domains are indeed inherently sequential (i.e., do not require concurrency). We suggest some real-world domains with required concurrency, and use a compilation argument to show that these are harder in the sense that they correspond to longer sequential plans. We conclude with the argument that temporal planners should be evaluated on both inherently sequential domains as well as those requiring concurrency.