TY - JOUR
T1 - Unsupervised Learning of Optical Flow with CNN-Based Non-Local Filtering
AU - Tian, Long
AU - Tu, Zhigang
AU - Zhang, Dejun
AU - Liu, Jun
AU - Li, Baoxin
AU - Yuan, Junsong
N1 - Funding Information:
Manuscript received January 15, 2020; revised June 1, 2020 and July 13, 2020; accepted July 27, 2020. Date of publication August 5, 2020; date of current version August 20, 2020. This work was supported in part by the National Key Research and Development Program of China under Grant 2018YFB2100603, in part by the Wuhan University-Infinova Project under Grant 2019010019, and in part by the Natural Science Fund of Hubei Province under Grant 2017CFB598. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Francesco G. B. De Natale. (Long Tian and Zhigang Tu are co-first authors.) (Corresponding author: Zhigang Tu.) Zhigang Tu is with the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China (e-mail: tuzhigang@whu.edu.cn). Long Tian resides in Chengdu, China (e-mail: 365278276@qq.com).
Publisher Copyright:
© 1992-2012 IEEE.
PY - 2020
Y1 - 2020
N2 - Estimating optical flow from successive video frames is one of the fundamental problems in computer vision and image processing. In the era of deep learning, many methods have been proposed to use convolutional neural networks (CNNs) for optical flow estimation in an unsupervised manner. However, the performance of unsupervised optical flow approaches is still unsatisfactory and often lagging far behind their supervised counterparts, primarily due to over-smoothing across motion boundaries and occlusion. To address these issues, in this paper, we propose a novel method with a new post-processing term and an effective loss function to estimate optical flow in an unsupervised, end-to-end learning manner. Specifically, we first exploit a CNN-based non-local term to refine the estimated optical flow by removing noise and decreasing blur around motion boundaries. This is implemented via automatically learning weights of dependencies over a large spatial neighborhood. Because of its learning ability, the method is effective for various complicated image sequences. Secondly, to reduce the influence of occlusion, a symmetrical energy formulation is introduced to detect the occlusion map from refined bi-directional optical flows. Then the occlusion map is integrated to the loss function. Extensive experiments are conducted on challenging datasets, i.e. FlyingChairs, MPI-Sintel and KITTI to evaluate the performance of the proposed method. The state-of-the-art results demonstrate the effectiveness of our proposed method.
AB - Estimating optical flow from successive video frames is one of the fundamental problems in computer vision and image processing. In the era of deep learning, many methods have been proposed to use convolutional neural networks (CNNs) for optical flow estimation in an unsupervised manner. However, the performance of unsupervised optical flow approaches is still unsatisfactory and often lagging far behind their supervised counterparts, primarily due to over-smoothing across motion boundaries and occlusion. To address these issues, in this paper, we propose a novel method with a new post-processing term and an effective loss function to estimate optical flow in an unsupervised, end-to-end learning manner. Specifically, we first exploit a CNN-based non-local term to refine the estimated optical flow by removing noise and decreasing blur around motion boundaries. This is implemented via automatically learning weights of dependencies over a large spatial neighborhood. Because of its learning ability, the method is effective for various complicated image sequences. Secondly, to reduce the influence of occlusion, a symmetrical energy formulation is introduced to detect the occlusion map from refined bi-directional optical flows. Then the occlusion map is integrated to the loss function. Extensive experiments are conducted on challenging datasets, i.e. FlyingChairs, MPI-Sintel and KITTI to evaluate the performance of the proposed method. The state-of-the-art results demonstrate the effectiveness of our proposed method.
KW - Optical flow
KW - loss function
KW - non-local term
KW - occlusion map
KW - unsupervised learning
UR - http://www.scopus.com/inward/record.url?scp=85090788738&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85090788738&partnerID=8YFLogxK
U2 - 10.1109/TIP.2020.3013168
DO - 10.1109/TIP.2020.3013168
M3 - Article
AN - SCOPUS:85090788738
SN - 1057-7149
VL - 29
SP - 8429
EP - 8442
JO - IEEE Transactions on Image Processing
JF - IEEE Transactions on Image Processing
M1 - 9159910
ER -