In this paper, a SELective Perceptual-based (SELP) scheme is presented to reduce the complexity of popular super-resolution (SR) algorithms while maintaining the desired quality of the enhanced images/video. A perceptual Human Visual System (HVS) model is proposed to compute the contrast sensitivity threshold for a given background intensity. The obtained thresholds are used to select which pixels are super-resolved based on the perceived visibility of local edges. This is accomplished by estimating the contrast sensitivity threshold locally over a block. Next, the absolute difference between each pixel and its neighbors is computed and compared to the threshold upon which a decision is made to include the pixel in the SR estimator for the next iteration or not. The perceptual model is integrated into a MAP-based SR algorithm as well as a fast ML estimator. Simulation results show up to 47% reduction on average in computational complexity with comparable SNR gains and visual quality.