Coral reef mapping is an indispensable step in coral conservation efforts across the globe. Monitoring reefs at regular intervals helps conservationists understand and address the problems causing coral reef degradation. Autonomous Underwater Vehicles (AUVs) have a tremendous potential to assist humans in these efforts. Delegating mapping and measurement acquisition tasks to AUVs would not only limit the number of human divers required for the missions but could also improve the quality of the maps developed. Consistency in imagery and spectroscopic measurements could be significantly improved by keeping the imagery payload at a fixed distance from the reefs to reduce heteroscedasticity in the measurements. To this end, we present a Terrain-Relative Diver Following system for an AUV that can follow a human diver while maintaining a fixed distance from the terrain. Our proposed system consists of separate modules for diver detection, tracking, and terrain following. We extensively tested our system in Gazebo simulation environment with three different terrain models, including a terrain model of a coral reef in Honaunau Bay, Hawaii. To the best of our knowledge, this is the first diver following system that also carries out terrain-relative navigation, ensuring minimal variation of distance to the terrain. We have released the code for our system, and the datasets used in the detection module.