This paper addresses the issue of tracking partially occluded targets in videos recorded by moving cameras of either handhold or airborne. We propose a fast geometric constraint global motion algorithm to reduce the computation overhead dramatically and the effect caused by outliers from moving targets. A recursive leastsquares filter with forgetting factor is utilized to filter out disturbances and to provide a better estimation of the target's position in the current frame as well as the prediction of the position and velocity for the next frame. The filter uses the affine model and the primary search result to construct a kinetic model. After that, a compact search region is formed based on the prediction to reduce mismatch and improve computation speed. The adaptive template matching is applied to improve the performance further. With these important steps, a tracking algorithm is developed and tested on real video sequences.