We propose a method for defect detection based on taking the sign information of Walsh Hadamard Transform (WHT) coefficients. The core of the proposed algorithm involves only three steps that can all be implemented very efficiently: applying the forward WHT, taking the sign of the transform coefficients, and taking an inverse WHT using only the sign information. Our implementation takes only 7 milliseconds for a 512 × 512 image on a PC platform. As a result, the proposed method is more efficient than the PHase Only Transform (PHOT) method and other methods in literature. In addition, the proposed approach is capable of detecting defects of varying shapes, by combining the 2-dimensional WHT and 1-dimensional WHT; and can detect defects in images with strong object boundaries by utilizing a reference image. The proposed algorithm is robust over different background image patterns and varying illumination conditions. We evaluated the proposed method both visually and quantitatively and obtained good results on images from various defect detection applications.