Counting Apples and Oranges with Deep Learning: A Data-Driven Approach

Steven W. Chen, Shreyas S. Shivakumar, Sandeep Dcunha, Jnaneshwar Das, Edidiong Okon, Chao Qu, Camillo J. Taylor, Vijay Kumar

Research output: Contribution to journalArticle

31 Citations (Scopus)

Abstract

This paper describes a fruit counting pipeline based on deep learning that accurately counts fruit in unstructured environments. Obtaining reliable fruit counts is challenging because of variations in appearance due to illumination changes and occlusions from foliage and neighboring fruits. We propose a novel approach that uses deep learning to map from input images to total fruit counts. The pipeline utilizes a custom crowdsourcing platform to quickly label large data sets. A blob detector based on a fully convolutional network extracts candidate regions in the images. A counting algorithm based on a second convolutional network then estimates the number of fruits in each region. Finally, a linear regression model maps that fruit count estimate to a final fruit count. We analyze the performance of the pipeline on two distinct data sets of oranges in daylight, and green apples at night, utilizing human generated labels as ground truth. We also show that the pipeline has a short training time and performs well with a limited data set size. Our method generalizes across both data sets and is able to perform well even on highly occluded fruits that are challenging for human labelers to annotate.

Original languageEnglish (US)
Article number7814145
Pages (from-to)781-788
Number of pages8
JournalIEEE Robotics and Automation Letters
Volume2
Issue number2
DOIs
StatePublished - Apr 1 2017

Fingerprint

Apple
Fruit
Fruits
Data-driven
Counting
Count
Pipelines
Labels
Learning
Deep learning
Linear Regression Model
Occlusion
Linear regression
Large Data Sets
Estimate
Illumination
Lighting
Detector
Detectors
Distinct

Keywords

  • Agricultural automation
  • categorization
  • object detection
  • segmentation
  • Visual learning

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Human-Computer Interaction
  • Biomedical Engineering
  • Mechanical Engineering
  • Control and Optimization
  • Artificial Intelligence
  • Computer Science Applications
  • Computer Vision and Pattern Recognition

Cite this

Chen, S. W., Shivakumar, S. S., Dcunha, S., Das, J., Okon, E., Qu, C., ... Kumar, V. (2017). Counting Apples and Oranges with Deep Learning: A Data-Driven Approach. IEEE Robotics and Automation Letters, 2(2), 781-788. [7814145]. https://doi.org/10.1109/LRA.2017.2651944

Counting Apples and Oranges with Deep Learning : A Data-Driven Approach. / Chen, Steven W.; Shivakumar, Shreyas S.; Dcunha, Sandeep; Das, Jnaneshwar; Okon, Edidiong; Qu, Chao; Taylor, Camillo J.; Kumar, Vijay.

In: IEEE Robotics and Automation Letters, Vol. 2, No. 2, 7814145, 01.04.2017, p. 781-788.

Research output: Contribution to journalArticle

Chen, SW, Shivakumar, SS, Dcunha, S, Das, J, Okon, E, Qu, C, Taylor, CJ & Kumar, V 2017, 'Counting Apples and Oranges with Deep Learning: A Data-Driven Approach' IEEE Robotics and Automation Letters, vol. 2, no. 2, 7814145, pp. 781-788. https://doi.org/10.1109/LRA.2017.2651944
Chen, Steven W. ; Shivakumar, Shreyas S. ; Dcunha, Sandeep ; Das, Jnaneshwar ; Okon, Edidiong ; Qu, Chao ; Taylor, Camillo J. ; Kumar, Vijay. / Counting Apples and Oranges with Deep Learning : A Data-Driven Approach. In: IEEE Robotics and Automation Letters. 2017 ; Vol. 2, No. 2. pp. 781-788.
@article{8dd2150b77594863b2b1a78850ee8354,
title = "Counting Apples and Oranges with Deep Learning: A Data-Driven Approach",
abstract = "This paper describes a fruit counting pipeline based on deep learning that accurately counts fruit in unstructured environments. Obtaining reliable fruit counts is challenging because of variations in appearance due to illumination changes and occlusions from foliage and neighboring fruits. We propose a novel approach that uses deep learning to map from input images to total fruit counts. The pipeline utilizes a custom crowdsourcing platform to quickly label large data sets. A blob detector based on a fully convolutional network extracts candidate regions in the images. A counting algorithm based on a second convolutional network then estimates the number of fruits in each region. Finally, a linear regression model maps that fruit count estimate to a final fruit count. We analyze the performance of the pipeline on two distinct data sets of oranges in daylight, and green apples at night, utilizing human generated labels as ground truth. We also show that the pipeline has a short training time and performs well with a limited data set size. Our method generalizes across both data sets and is able to perform well even on highly occluded fruits that are challenging for human labelers to annotate.",
keywords = "Agricultural automation, categorization, object detection, segmentation, Visual learning",
author = "Chen, {Steven W.} and Shivakumar, {Shreyas S.} and Sandeep Dcunha and Jnaneshwar Das and Edidiong Okon and Chao Qu and Taylor, {Camillo J.} and Vijay Kumar",
year = "2017",
month = "4",
day = "1",
doi = "10.1109/LRA.2017.2651944",
language = "English (US)",
volume = "2",
pages = "781--788",
journal = "IEEE Robotics and Automation Letters",
issn = "2377-3766",
number = "2",

}

TY - JOUR

T1 - Counting Apples and Oranges with Deep Learning

T2 - A Data-Driven Approach

AU - Chen, Steven W.

AU - Shivakumar, Shreyas S.

AU - Dcunha, Sandeep

AU - Das, Jnaneshwar

AU - Okon, Edidiong

AU - Qu, Chao

AU - Taylor, Camillo J.

AU - Kumar, Vijay

PY - 2017/4/1

Y1 - 2017/4/1

N2 - This paper describes a fruit counting pipeline based on deep learning that accurately counts fruit in unstructured environments. Obtaining reliable fruit counts is challenging because of variations in appearance due to illumination changes and occlusions from foliage and neighboring fruits. We propose a novel approach that uses deep learning to map from input images to total fruit counts. The pipeline utilizes a custom crowdsourcing platform to quickly label large data sets. A blob detector based on a fully convolutional network extracts candidate regions in the images. A counting algorithm based on a second convolutional network then estimates the number of fruits in each region. Finally, a linear regression model maps that fruit count estimate to a final fruit count. We analyze the performance of the pipeline on two distinct data sets of oranges in daylight, and green apples at night, utilizing human generated labels as ground truth. We also show that the pipeline has a short training time and performs well with a limited data set size. Our method generalizes across both data sets and is able to perform well even on highly occluded fruits that are challenging for human labelers to annotate.

AB - This paper describes a fruit counting pipeline based on deep learning that accurately counts fruit in unstructured environments. Obtaining reliable fruit counts is challenging because of variations in appearance due to illumination changes and occlusions from foliage and neighboring fruits. We propose a novel approach that uses deep learning to map from input images to total fruit counts. The pipeline utilizes a custom crowdsourcing platform to quickly label large data sets. A blob detector based on a fully convolutional network extracts candidate regions in the images. A counting algorithm based on a second convolutional network then estimates the number of fruits in each region. Finally, a linear regression model maps that fruit count estimate to a final fruit count. We analyze the performance of the pipeline on two distinct data sets of oranges in daylight, and green apples at night, utilizing human generated labels as ground truth. We also show that the pipeline has a short training time and performs well with a limited data set size. Our method generalizes across both data sets and is able to perform well even on highly occluded fruits that are challenging for human labelers to annotate.

KW - Agricultural automation

KW - categorization

KW - object detection

KW - segmentation

KW - Visual learning

UR - http://www.scopus.com/inward/record.url?scp=85028676918&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85028676918&partnerID=8YFLogxK

U2 - 10.1109/LRA.2017.2651944

DO - 10.1109/LRA.2017.2651944

M3 - Article

VL - 2

SP - 781

EP - 788

JO - IEEE Robotics and Automation Letters

JF - IEEE Robotics and Automation Letters

SN - 2377-3766

IS - 2

M1 - 7814145

ER -