This paper focuses on applying an interval recursive least-squares (RLS) filter to a video target tracking problem. An RLS filter can be sensitive to variations in filter parameters and disturbance to state observations to make the solutions impractical in practical problems. Specially, in the application of video target tracking using an RLS filter, inaccurate parameters in the affine model may result in noticeable deviations from true target positions to lose the target. To make results robust, each filter parameter and state observation is allowed to vary in an interval. Motivated by this idea, an interval RLS filter is proposed to produce state estimation and prediction by narrow intervals. Simulations show that an interval RLS filter is robust to state and observation noise and variations in filter parameters and state observations, and outperforms an interval Kalman filter. Using an interval RLS filter, a video target tracking algorithm is developed to estimate the target position in each frame. The proposed tracking algorithm using an interval RLS filter is robust to noise in video sequences and error of the affine models, and outperforms that using an RLS filter. Performance evaluations using real-world video sequences are provided to demonstrate effectiveness of the proposed algorithm.