Automated tracking of targets within outdoor infrared (IR) video sequences poses a host of challenges. These include automatic gain adjustment in the IR camera, extreme granularity, large luminance changes, and uncontrolled environmental factors such as moving foliage, animals, and birds, among others. To address these problems, we present an IR video target tracking system for stationary cameras that learns and divides the video frames into reliable and unreliable regions. A difference-frame-based method can recognize moving regions with high sensitivity and reliably discern background clutter from target motion. A low-complexity target validation process is presented, which in conjunction with the reliable region masking, dramatically reduces the number of false alarms. We demonstrate the outstanding performance of the proposed system using real-world IR video sequences with difficult background motion clutter, as well as with small and blurred moving targets.