Robust Fruit Counting: Combining Deep Learning, Tracking, and Structure from Motion

Xu Liu, Steven W. Chen, Shreyas Aditya, Nivedha Sivakumar, Sandeep Dcunha, Chao Qu, Camillo J. Taylor, Jnaneshwar Das, Vijay Kumar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Citations (Scopus)

Abstract

We present a novel fruit counting pipeline that combines deep segmentation, frame to frame tracking, and 3D localization to accurately count visible fruits across a sequence of images. Our pipeline works on image streams from a monocular camera, both in natural light, as well as with controlled illumination at night. We first train a Fully Convolutional Network (FCN) and segment video frame images into fruit and non-fruit pixels. We then track fruits across frames using the Hungarian Algorithm where the objective cost is determined from a Kalman Filter corrected Kanade-Lucas-Tomasi (KLT) Tracker. In order to correct the estimated count from tracking process, we combine tracking results with a Structure from Motion (SfM) algorithm to calculate relative 3D locations and size estimates to reject outliers and double counted fruit tracks. We evaluate our algorithm by comparing with ground-truth human-annotated visual counts. Our results demonstrate that our pipeline is able to accurately and reliably count fruits across image sequences, and the correction step can significantly improve the counting accuracy and robustness. Although discussed in the context of fruit counting, our work can extend to detection, tracking, and counting of a variety of other stationary features of interest such as leaf-spots, wilt, and blossom.

Original languageEnglish (US)
Title of host publication2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1045-1052
Number of pages8
ISBN (Electronic)9781538680940
DOIs
StatePublished - Dec 27 2018
Externally publishedYes
Event2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018 - Madrid, Spain
Duration: Oct 1 2018Oct 5 2018

Publication series

NameIEEE International Conference on Intelligent Robots and Systems
ISSN (Print)2153-0858
ISSN (Electronic)2153-0866

Conference

Conference2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018
CountrySpain
CityMadrid
Period10/1/1810/5/18

Fingerprint

Fruits
Pipelines
Deep learning
Kalman filters
Lighting
Pixels
Cameras
Costs

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Software
  • Computer Vision and Pattern Recognition
  • Computer Science Applications

Cite this

Liu, X., Chen, S. W., Aditya, S., Sivakumar, N., Dcunha, S., Qu, C., ... Kumar, V. (2018). Robust Fruit Counting: Combining Deep Learning, Tracking, and Structure from Motion. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018 (pp. 1045-1052). [8594239] (IEEE International Conference on Intelligent Robots and Systems). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IROS.2018.8594239

Robust Fruit Counting : Combining Deep Learning, Tracking, and Structure from Motion. / Liu, Xu; Chen, Steven W.; Aditya, Shreyas; Sivakumar, Nivedha; Dcunha, Sandeep; Qu, Chao; Taylor, Camillo J.; Das, Jnaneshwar; Kumar, Vijay.

2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 1045-1052 8594239 (IEEE International Conference on Intelligent Robots and Systems).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Liu, X, Chen, SW, Aditya, S, Sivakumar, N, Dcunha, S, Qu, C, Taylor, CJ, Das, J & Kumar, V 2018, Robust Fruit Counting: Combining Deep Learning, Tracking, and Structure from Motion. in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018., 8594239, IEEE International Conference on Intelligent Robots and Systems, Institute of Electrical and Electronics Engineers Inc., pp. 1045-1052, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018, Madrid, Spain, 10/1/18. https://doi.org/10.1109/IROS.2018.8594239
Liu X, Chen SW, Aditya S, Sivakumar N, Dcunha S, Qu C et al. Robust Fruit Counting: Combining Deep Learning, Tracking, and Structure from Motion. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 1045-1052. 8594239. (IEEE International Conference on Intelligent Robots and Systems). https://doi.org/10.1109/IROS.2018.8594239
Liu, Xu ; Chen, Steven W. ; Aditya, Shreyas ; Sivakumar, Nivedha ; Dcunha, Sandeep ; Qu, Chao ; Taylor, Camillo J. ; Das, Jnaneshwar ; Kumar, Vijay. / Robust Fruit Counting : Combining Deep Learning, Tracking, and Structure from Motion. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 1045-1052 (IEEE International Conference on Intelligent Robots and Systems).
@inproceedings{bdd12002f56f46ad99e09e518659f534,
title = "Robust Fruit Counting: Combining Deep Learning, Tracking, and Structure from Motion",
abstract = "We present a novel fruit counting pipeline that combines deep segmentation, frame to frame tracking, and 3D localization to accurately count visible fruits across a sequence of images. Our pipeline works on image streams from a monocular camera, both in natural light, as well as with controlled illumination at night. We first train a Fully Convolutional Network (FCN) and segment video frame images into fruit and non-fruit pixels. We then track fruits across frames using the Hungarian Algorithm where the objective cost is determined from a Kalman Filter corrected Kanade-Lucas-Tomasi (KLT) Tracker. In order to correct the estimated count from tracking process, we combine tracking results with a Structure from Motion (SfM) algorithm to calculate relative 3D locations and size estimates to reject outliers and double counted fruit tracks. We evaluate our algorithm by comparing with ground-truth human-annotated visual counts. Our results demonstrate that our pipeline is able to accurately and reliably count fruits across image sequences, and the correction step can significantly improve the counting accuracy and robustness. Although discussed in the context of fruit counting, our work can extend to detection, tracking, and counting of a variety of other stationary features of interest such as leaf-spots, wilt, and blossom.",
author = "Xu Liu and Chen, {Steven W.} and Shreyas Aditya and Nivedha Sivakumar and Sandeep Dcunha and Chao Qu and Taylor, {Camillo J.} and Jnaneshwar Das and Vijay Kumar",
year = "2018",
month = "12",
day = "27",
doi = "10.1109/IROS.2018.8594239",
language = "English (US)",
series = "IEEE International Conference on Intelligent Robots and Systems",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "1045--1052",
booktitle = "2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018",

}

TY - GEN

T1 - Robust Fruit Counting

T2 - Combining Deep Learning, Tracking, and Structure from Motion

AU - Liu, Xu

AU - Chen, Steven W.

AU - Aditya, Shreyas

AU - Sivakumar, Nivedha

AU - Dcunha, Sandeep

AU - Qu, Chao

AU - Taylor, Camillo J.

AU - Das, Jnaneshwar

AU - Kumar, Vijay

PY - 2018/12/27

Y1 - 2018/12/27

N2 - We present a novel fruit counting pipeline that combines deep segmentation, frame to frame tracking, and 3D localization to accurately count visible fruits across a sequence of images. Our pipeline works on image streams from a monocular camera, both in natural light, as well as with controlled illumination at night. We first train a Fully Convolutional Network (FCN) and segment video frame images into fruit and non-fruit pixels. We then track fruits across frames using the Hungarian Algorithm where the objective cost is determined from a Kalman Filter corrected Kanade-Lucas-Tomasi (KLT) Tracker. In order to correct the estimated count from tracking process, we combine tracking results with a Structure from Motion (SfM) algorithm to calculate relative 3D locations and size estimates to reject outliers and double counted fruit tracks. We evaluate our algorithm by comparing with ground-truth human-annotated visual counts. Our results demonstrate that our pipeline is able to accurately and reliably count fruits across image sequences, and the correction step can significantly improve the counting accuracy and robustness. Although discussed in the context of fruit counting, our work can extend to detection, tracking, and counting of a variety of other stationary features of interest such as leaf-spots, wilt, and blossom.

AB - We present a novel fruit counting pipeline that combines deep segmentation, frame to frame tracking, and 3D localization to accurately count visible fruits across a sequence of images. Our pipeline works on image streams from a monocular camera, both in natural light, as well as with controlled illumination at night. We first train a Fully Convolutional Network (FCN) and segment video frame images into fruit and non-fruit pixels. We then track fruits across frames using the Hungarian Algorithm where the objective cost is determined from a Kalman Filter corrected Kanade-Lucas-Tomasi (KLT) Tracker. In order to correct the estimated count from tracking process, we combine tracking results with a Structure from Motion (SfM) algorithm to calculate relative 3D locations and size estimates to reject outliers and double counted fruit tracks. We evaluate our algorithm by comparing with ground-truth human-annotated visual counts. Our results demonstrate that our pipeline is able to accurately and reliably count fruits across image sequences, and the correction step can significantly improve the counting accuracy and robustness. Although discussed in the context of fruit counting, our work can extend to detection, tracking, and counting of a variety of other stationary features of interest such as leaf-spots, wilt, and blossom.

UR - http://www.scopus.com/inward/record.url?scp=85062988241&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062988241&partnerID=8YFLogxK

U2 - 10.1109/IROS.2018.8594239

DO - 10.1109/IROS.2018.8594239

M3 - Conference contribution

AN - SCOPUS:85062988241

T3 - IEEE International Conference on Intelligent Robots and Systems

SP - 1045

EP - 1052

BT - 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -