Towards scalable application checkpointing with parallel file system delegation

The ever-increasing scale of modern highperformance computing (HPC) systems presents a variety of challenges to the parallel file system (PFS) based storage in these systems. The scalability of application checkpointing is a particularly important challenge because it is critical to the reliability of computing and it often dominates the I/Os in a HPC system. When a large number of parallel processes simultaneously perform checkpointing, the PFS metadata servers can become a serious bottleneck due to the large volume of concurrent metadata operations. This paper specifically addresses this PFS metadata management issue in order to support scalable application checkpointing in large HPC systems. It proposes a new technique named PFS-delegation which delegates the management of the PFS storage space used for checkpointing to applications, thereby relieving the load of metadata operations on the PFS during their checkpointing. This proposed technique is prototyped on PVFS2, a widely used PFS implementation, and evaluated on a HPC cluster using a representative parallel I/O benchmark, IOR. Experiments with up to 128 parallel processes show that the PFS-delegation based checkpointing is significantly faster than the traditional shared-file and file-per-process based checkpointing methods (7% and 10% speedup when the underlying PVFS2 uses a centralized metadata server; 22% and 31% speedup when using distributed metadata servers). The results also demonstrate that the PFS-delegation based checkpointing substantially reduces the total number of metadata operations handled by the metadata servers during the checkpointing.