System comparison procedures for automatic target recognition systems

Anne E. Catlin, Kenneth W. Bauer, Edward F. Mykytka, Douglas Montgomery

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Estimating the performance of an automatic target recognition (ATR) system in terms of its probability of successfully identifying a target involves extensive image collection and processing, which can be very time-consuming and expensive. Therefore, we investigate the Wald sequential test for the difference in two proportions as a sample size-reducing alternative to ranking and selection procedures and confidence intervals. An analysis of the test parameters leads to a practical methodology for implementing the Wald test for fairly comparing two systems, based on specific experimental goals. The test is also extended with the modified, sequentially rejective Bonferroni procedure for the multiple pairwise comparison of more than two systems. Two sampling schemes for different experimental goals are also discussed. The test methodology is applied to actual data to compare different configurations of a specific ATR system, with the results demonstrating that the modified Wald sequential procedure is a useful alternative to comparing proportions with confidence intervals, particularly when data are expensive.

Original languageEnglish (US)
Pages (from-to)357-371
Number of pages15
JournalNaval Research Logistics
Volume46
Issue number4
DOIs
StatePublished - Jun 1999

Fingerprint

Automatic target recognition
Target Recognition
Wald Test
Confidence interval
Proportion
Sampling
Ranking and Selection
Sequential Test
Bonferroni
Sequential Procedure
Multiple Comparisons
Methodology
Pairwise Comparisons
Alternatives
Selection Procedures
Processing
Sample Size
Configuration
Target

ASJC Scopus subject areas

  • Management Science and Operations Research

Cite this

System comparison procedures for automatic target recognition systems. / Catlin, Anne E.; Bauer, Kenneth W.; Mykytka, Edward F.; Montgomery, Douglas.

In: Naval Research Logistics, Vol. 46, No. 4, 06.1999, p. 357-371.

Research output: Contribution to journalArticle

Catlin, Anne E. ; Bauer, Kenneth W. ; Mykytka, Edward F. ; Montgomery, Douglas. / System comparison procedures for automatic target recognition systems. In: Naval Research Logistics. 1999 ; Vol. 46, No. 4. pp. 357-371.
@article{abd6a8e8e9884d368cf28da71666a7cd,
title = "System comparison procedures for automatic target recognition systems",
abstract = "Estimating the performance of an automatic target recognition (ATR) system in terms of its probability of successfully identifying a target involves extensive image collection and processing, which can be very time-consuming and expensive. Therefore, we investigate the Wald sequential test for the difference in two proportions as a sample size-reducing alternative to ranking and selection procedures and confidence intervals. An analysis of the test parameters leads to a practical methodology for implementing the Wald test for fairly comparing two systems, based on specific experimental goals. The test is also extended with the modified, sequentially rejective Bonferroni procedure for the multiple pairwise comparison of more than two systems. Two sampling schemes for different experimental goals are also discussed. The test methodology is applied to actual data to compare different configurations of a specific ATR system, with the results demonstrating that the modified Wald sequential procedure is a useful alternative to comparing proportions with confidence intervals, particularly when data are expensive.",
author = "Catlin, {Anne E.} and Bauer, {Kenneth W.} and Mykytka, {Edward F.} and Douglas Montgomery",
year = "1999",
month = "6",
doi = "10.1002/(SICI)1520-6750(199906)46:4<357::AID-NAV2>3.0.CO;2-P",
language = "English (US)",
volume = "46",
pages = "357--371",
journal = "Naval Research Logistics",
issn = "0894-069X",
publisher = "John Wiley and Sons Inc.",
number = "4",

}

TY - JOUR

T1 - System comparison procedures for automatic target recognition systems

AU - Catlin, Anne E.

AU - Bauer, Kenneth W.

AU - Mykytka, Edward F.

AU - Montgomery, Douglas

PY - 1999/6

Y1 - 1999/6

N2 - Estimating the performance of an automatic target recognition (ATR) system in terms of its probability of successfully identifying a target involves extensive image collection and processing, which can be very time-consuming and expensive. Therefore, we investigate the Wald sequential test for the difference in two proportions as a sample size-reducing alternative to ranking and selection procedures and confidence intervals. An analysis of the test parameters leads to a practical methodology for implementing the Wald test for fairly comparing two systems, based on specific experimental goals. The test is also extended with the modified, sequentially rejective Bonferroni procedure for the multiple pairwise comparison of more than two systems. Two sampling schemes for different experimental goals are also discussed. The test methodology is applied to actual data to compare different configurations of a specific ATR system, with the results demonstrating that the modified Wald sequential procedure is a useful alternative to comparing proportions with confidence intervals, particularly when data are expensive.

AB - Estimating the performance of an automatic target recognition (ATR) system in terms of its probability of successfully identifying a target involves extensive image collection and processing, which can be very time-consuming and expensive. Therefore, we investigate the Wald sequential test for the difference in two proportions as a sample size-reducing alternative to ranking and selection procedures and confidence intervals. An analysis of the test parameters leads to a practical methodology for implementing the Wald test for fairly comparing two systems, based on specific experimental goals. The test is also extended with the modified, sequentially rejective Bonferroni procedure for the multiple pairwise comparison of more than two systems. Two sampling schemes for different experimental goals are also discussed. The test methodology is applied to actual data to compare different configurations of a specific ATR system, with the results demonstrating that the modified Wald sequential procedure is a useful alternative to comparing proportions with confidence intervals, particularly when data are expensive.

UR - http://www.scopus.com/inward/record.url?scp=0033149366&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0033149366&partnerID=8YFLogxK

U2 - 10.1002/(SICI)1520-6750(199906)46:4<357::AID-NAV2>3.0.CO;2-P

DO - 10.1002/(SICI)1520-6750(199906)46:4<357::AID-NAV2>3.0.CO;2-P

M3 - Article

VL - 46

SP - 357

EP - 371

JO - Naval Research Logistics

JF - Naval Research Logistics

SN - 0894-069X

IS - 4

ER -