Monocular Camera Based Fruit Counting and Mapping with Semantic Data Association

Xu Liu, Steven W. Chen, Chenhao Liu, Shreyas S. Shivakumar, Jnaneshwar Das, Camillo J. Taylor, James Underwood, Vijay Kumar

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

In this letter, we present a cheap, lightweight, and fast fruit counting pipeline. Our pipeline relies only on a monocular camera, and achieves counting performance comparable to a state-of-the-art fruit counting system that utilizes an expensive sensor suite including a monocular camera, LiDAR and GPS/INS on a mango dataset. Our pipeline begins with a fruit and tree trunk detection component that uses state-of-the-art convolutional neural networks (CNNs). It then tracks fruits and tree trunks across images, with a Kalman Filter fusing measurements from the CNN detectors and an optical flow estimator. Finally, fruit count and map are estimated by an efficient fruit-as-feature semantic structure from motion algorithm that converts two-dimensional (2-D) tracks of fruits and trunks into 3-D landmarks, and uses these landmarks to identify double counting scenarios. There are many benefits of developing such a low cost and lightweight fruit counting system, including applicability to agriculture in developing countries, where monetary constraints or unstructured environments necessitate cheaper hardware solutions.

Original languageEnglish (US)
Article number8653965
Pages (from-to)2296-2303
Number of pages8
JournalIEEE Robotics and Automation Letters
Volume4
Issue number3
DOIs
StatePublished - Jul 1 2019

Fingerprint

Data Association
Fruit
Fruits
Counting
Camera
Semantics
Cameras
Pipelines
Landmarks
Neural Networks
Neural networks
Structure from Motion
Optical flows
Optical Flow
Lidar
Developing Countries
Agriculture
Developing countries
Kalman filters
Kalman Filter

Keywords

  • deep learning in robotics and automation
  • mapping
  • object detection
  • Robotics in agriculture and forestry
  • segmentation and categorization
  • visual tracking

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Human-Computer Interaction
  • Biomedical Engineering
  • Mechanical Engineering
  • Control and Optimization
  • Artificial Intelligence
  • Computer Science Applications
  • Computer Vision and Pattern Recognition

Cite this

Liu, X., Chen, S. W., Liu, C., Shivakumar, S. S., Das, J., Taylor, C. J., ... Kumar, V. (2019). Monocular Camera Based Fruit Counting and Mapping with Semantic Data Association. IEEE Robotics and Automation Letters, 4(3), 2296-2303. [8653965]. https://doi.org/10.1109/LRA.2019.2901987

Monocular Camera Based Fruit Counting and Mapping with Semantic Data Association. / Liu, Xu; Chen, Steven W.; Liu, Chenhao; Shivakumar, Shreyas S.; Das, Jnaneshwar; Taylor, Camillo J.; Underwood, James; Kumar, Vijay.

In: IEEE Robotics and Automation Letters, Vol. 4, No. 3, 8653965, 01.07.2019, p. 2296-2303.

Research output: Contribution to journalArticle

Liu, X, Chen, SW, Liu, C, Shivakumar, SS, Das, J, Taylor, CJ, Underwood, J & Kumar, V 2019, 'Monocular Camera Based Fruit Counting and Mapping with Semantic Data Association', IEEE Robotics and Automation Letters, vol. 4, no. 3, 8653965, pp. 2296-2303. https://doi.org/10.1109/LRA.2019.2901987
Liu, Xu ; Chen, Steven W. ; Liu, Chenhao ; Shivakumar, Shreyas S. ; Das, Jnaneshwar ; Taylor, Camillo J. ; Underwood, James ; Kumar, Vijay. / Monocular Camera Based Fruit Counting and Mapping with Semantic Data Association. In: IEEE Robotics and Automation Letters. 2019 ; Vol. 4, No. 3. pp. 2296-2303.
@article{8df7f88ce7524f4a91bf502781c218f3,
title = "Monocular Camera Based Fruit Counting and Mapping with Semantic Data Association",
abstract = "In this letter, we present a cheap, lightweight, and fast fruit counting pipeline. Our pipeline relies only on a monocular camera, and achieves counting performance comparable to a state-of-the-art fruit counting system that utilizes an expensive sensor suite including a monocular camera, LiDAR and GPS/INS on a mango dataset. Our pipeline begins with a fruit and tree trunk detection component that uses state-of-the-art convolutional neural networks (CNNs). It then tracks fruits and tree trunks across images, with a Kalman Filter fusing measurements from the CNN detectors and an optical flow estimator. Finally, fruit count and map are estimated by an efficient fruit-as-feature semantic structure from motion algorithm that converts two-dimensional (2-D) tracks of fruits and trunks into 3-D landmarks, and uses these landmarks to identify double counting scenarios. There are many benefits of developing such a low cost and lightweight fruit counting system, including applicability to agriculture in developing countries, where monetary constraints or unstructured environments necessitate cheaper hardware solutions.",
keywords = "deep learning in robotics and automation, mapping, object detection, Robotics in agriculture and forestry, segmentation and categorization, visual tracking",
author = "Xu Liu and Chen, {Steven W.} and Chenhao Liu and Shivakumar, {Shreyas S.} and Jnaneshwar Das and Taylor, {Camillo J.} and James Underwood and Vijay Kumar",
year = "2019",
month = "7",
day = "1",
doi = "10.1109/LRA.2019.2901987",
language = "English (US)",
volume = "4",
pages = "2296--2303",
journal = "IEEE Robotics and Automation Letters",
issn = "2377-3766",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "3",

}

TY - JOUR

T1 - Monocular Camera Based Fruit Counting and Mapping with Semantic Data Association

AU - Liu, Xu

AU - Chen, Steven W.

AU - Liu, Chenhao

AU - Shivakumar, Shreyas S.

AU - Das, Jnaneshwar

AU - Taylor, Camillo J.

AU - Underwood, James

AU - Kumar, Vijay

PY - 2019/7/1

Y1 - 2019/7/1

N2 - In this letter, we present a cheap, lightweight, and fast fruit counting pipeline. Our pipeline relies only on a monocular camera, and achieves counting performance comparable to a state-of-the-art fruit counting system that utilizes an expensive sensor suite including a monocular camera, LiDAR and GPS/INS on a mango dataset. Our pipeline begins with a fruit and tree trunk detection component that uses state-of-the-art convolutional neural networks (CNNs). It then tracks fruits and tree trunks across images, with a Kalman Filter fusing measurements from the CNN detectors and an optical flow estimator. Finally, fruit count and map are estimated by an efficient fruit-as-feature semantic structure from motion algorithm that converts two-dimensional (2-D) tracks of fruits and trunks into 3-D landmarks, and uses these landmarks to identify double counting scenarios. There are many benefits of developing such a low cost and lightweight fruit counting system, including applicability to agriculture in developing countries, where monetary constraints or unstructured environments necessitate cheaper hardware solutions.

AB - In this letter, we present a cheap, lightweight, and fast fruit counting pipeline. Our pipeline relies only on a monocular camera, and achieves counting performance comparable to a state-of-the-art fruit counting system that utilizes an expensive sensor suite including a monocular camera, LiDAR and GPS/INS on a mango dataset. Our pipeline begins with a fruit and tree trunk detection component that uses state-of-the-art convolutional neural networks (CNNs). It then tracks fruits and tree trunks across images, with a Kalman Filter fusing measurements from the CNN detectors and an optical flow estimator. Finally, fruit count and map are estimated by an efficient fruit-as-feature semantic structure from motion algorithm that converts two-dimensional (2-D) tracks of fruits and trunks into 3-D landmarks, and uses these landmarks to identify double counting scenarios. There are many benefits of developing such a low cost and lightweight fruit counting system, including applicability to agriculture in developing countries, where monetary constraints or unstructured environments necessitate cheaper hardware solutions.

KW - deep learning in robotics and automation

KW - mapping

KW - object detection

KW - Robotics in agriculture and forestry

KW - segmentation and categorization

KW - visual tracking

UR - http://www.scopus.com/inward/record.url?scp=85063575627&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063575627&partnerID=8YFLogxK

U2 - 10.1109/LRA.2019.2901987

DO - 10.1109/LRA.2019.2901987

M3 - Article

VL - 4

SP - 2296

EP - 2303

JO - IEEE Robotics and Automation Letters

JF - IEEE Robotics and Automation Letters

SN - 2377-3766

IS - 3

M1 - 8653965

ER -