In this paper, a framework for validating any generic face detection algorithm's result is proposed. A two stage cascaded face validation filter is described that relies on a skin-color detector and on a face silhouette structure modeler towards increasing face detection capacity of any face detection algorithm. While the skin-color detector combines a static skin-color and a dynamic background-color modeler, the face silhouette structure modeler incorporates an aggregate of random field models combined through a Demspter-Shafer framework of evidence merging. Together, the two modelers validate any face subimage generated by face detection algorithms. Experiments conducted on FERET and on an in-house face database supports the claimfor improved face detection results using the proposed filter. An extension of the same framework towards head pose estimation is also suggested.