Interactively test driving an object detector: Estimating performance on unlabeled data

Rushil Anirudh, Pavan Turaga

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

In this paper, we study the problem of 'test-driving' a detector, i.e. allowing a human user to get a quick sense of how well the detector generalizes to their specific requirement. To this end, we present the first system that estimates detector performance interactively without extensive ground truthing using a human in the loop. We approach this as a problem of estimating proportions and show that it is possible to make accurate inferences on the proportion of classes or groups within a large data collection by observing only 5 - 10% of samples from the data. In estimating the false detections (for precision), the samples are chosen carefully such that the overall characteristics of the data collection are preserved. Next, inspired by its use in estimating disease propagation we apply pooled testing approaches to estimate missed detections (for recall) from the dataset. The estimates thus obtained are close to the ones obtained using ground truth, thus reducing the need for extensive labeling which is expensive and time consuming.

Original languageEnglish (US)
Title of host publication2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014
PublisherIEEE Computer Society
Pages175-182
Number of pages8
ISBN (Print)9781479949854
DOIs
StatePublished - 2014
Event2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014 - Steamboat Springs, CO, United States
Duration: Mar 24 2014Mar 26 2014

Other

Other2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014
CountryUnited States
CitySteamboat Springs, CO
Period3/24/143/26/14

Fingerprint

Detectors
Labeling
Testing

ASJC Scopus subject areas

  • Computer Science Applications
  • Computer Vision and Pattern Recognition

Cite this

Anirudh, R., & Turaga, P. (2014). Interactively test driving an object detector: Estimating performance on unlabeled data. In 2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014 (pp. 175-182). [6836104] IEEE Computer Society. https://doi.org/10.1109/WACV.2014.6836104

Interactively test driving an object detector : Estimating performance on unlabeled data. / Anirudh, Rushil; Turaga, Pavan.

2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014. IEEE Computer Society, 2014. p. 175-182 6836104.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Anirudh, R & Turaga, P 2014, Interactively test driving an object detector: Estimating performance on unlabeled data. in 2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014., 6836104, IEEE Computer Society, pp. 175-182, 2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014, Steamboat Springs, CO, United States, 3/24/14. https://doi.org/10.1109/WACV.2014.6836104
Anirudh R, Turaga P. Interactively test driving an object detector: Estimating performance on unlabeled data. In 2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014. IEEE Computer Society. 2014. p. 175-182. 6836104 https://doi.org/10.1109/WACV.2014.6836104
Anirudh, Rushil ; Turaga, Pavan. / Interactively test driving an object detector : Estimating performance on unlabeled data. 2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014. IEEE Computer Society, 2014. pp. 175-182
@inproceedings{f8961cb3598a4af88ac3ead4f754e85b,
title = "Interactively test driving an object detector: Estimating performance on unlabeled data",
abstract = "In this paper, we study the problem of 'test-driving' a detector, i.e. allowing a human user to get a quick sense of how well the detector generalizes to their specific requirement. To this end, we present the first system that estimates detector performance interactively without extensive ground truthing using a human in the loop. We approach this as a problem of estimating proportions and show that it is possible to make accurate inferences on the proportion of classes or groups within a large data collection by observing only 5 - 10{\%} of samples from the data. In estimating the false detections (for precision), the samples are chosen carefully such that the overall characteristics of the data collection are preserved. Next, inspired by its use in estimating disease propagation we apply pooled testing approaches to estimate missed detections (for recall) from the dataset. The estimates thus obtained are close to the ones obtained using ground truth, thus reducing the need for extensive labeling which is expensive and time consuming.",
author = "Rushil Anirudh and Pavan Turaga",
year = "2014",
doi = "10.1109/WACV.2014.6836104",
language = "English (US)",
isbn = "9781479949854",
pages = "175--182",
booktitle = "2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014",
publisher = "IEEE Computer Society",

}

TY - GEN

T1 - Interactively test driving an object detector

T2 - Estimating performance on unlabeled data

AU - Anirudh, Rushil

AU - Turaga, Pavan

PY - 2014

Y1 - 2014

N2 - In this paper, we study the problem of 'test-driving' a detector, i.e. allowing a human user to get a quick sense of how well the detector generalizes to their specific requirement. To this end, we present the first system that estimates detector performance interactively without extensive ground truthing using a human in the loop. We approach this as a problem of estimating proportions and show that it is possible to make accurate inferences on the proportion of classes or groups within a large data collection by observing only 5 - 10% of samples from the data. In estimating the false detections (for precision), the samples are chosen carefully such that the overall characteristics of the data collection are preserved. Next, inspired by its use in estimating disease propagation we apply pooled testing approaches to estimate missed detections (for recall) from the dataset. The estimates thus obtained are close to the ones obtained using ground truth, thus reducing the need for extensive labeling which is expensive and time consuming.

AB - In this paper, we study the problem of 'test-driving' a detector, i.e. allowing a human user to get a quick sense of how well the detector generalizes to their specific requirement. To this end, we present the first system that estimates detector performance interactively without extensive ground truthing using a human in the loop. We approach this as a problem of estimating proportions and show that it is possible to make accurate inferences on the proportion of classes or groups within a large data collection by observing only 5 - 10% of samples from the data. In estimating the false detections (for precision), the samples are chosen carefully such that the overall characteristics of the data collection are preserved. Next, inspired by its use in estimating disease propagation we apply pooled testing approaches to estimate missed detections (for recall) from the dataset. The estimates thus obtained are close to the ones obtained using ground truth, thus reducing the need for extensive labeling which is expensive and time consuming.

UR - http://www.scopus.com/inward/record.url?scp=84904609464&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84904609464&partnerID=8YFLogxK

U2 - 10.1109/WACV.2014.6836104

DO - 10.1109/WACV.2014.6836104

M3 - Conference contribution

AN - SCOPUS:84904609464

SN - 9781479949854

SP - 175

EP - 182

BT - 2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014

PB - IEEE Computer Society

ER -