Software performance testing usina covering arrays: Efficient screening designs with categorical factors

Research output: Chapter in Book/Report/Conference proceedingConference contribution

17 Citations (Scopus)

Abstract

Classical Design of Experiment (DOE) techniques have been in use for many years to aid in the performance testing of systems. In particular fractional factorial designs have been used in cases with many numerical factors to reduce the number of experimental runs necessary. For experiments involving categorical factors, this is not the case; experimenters regularly resort to exhaustive (full factorial) experiments. Recently, D-optimal designs have been used to reduce numbers of tests for experiments involving categorical factors because of their flexibility, but not necessarily because they can closely approximate full factorial results. In commonly used statistical packages, the only generic alternative for reduced experiments involving categorical factors is afforded by optimal designs. The extent to which D-optimal designs succeed in estimating exhaustive results has not been evaluated, and it is natural to determine this. An alternative design based on covering arrays may offer a better approximation of full factorial data. Covering arrays are used in software testing for accurate coverage of interactions, while D-optimal and factorial designs measure the amount of interaction. Initial work involved exhaustive generation of designs in order to compare covering arrays and D-optimal designs in approximating full factorial designs. In that setting, covering arrays provided better approximations of full factorial analysis.while ensuring coverage of all small interactions. Here we examine commercially viable covering array and D-optimal design generators to compare designs. Commercial covering array generators, while not as good as exhaustively generated designs, remain competitive with D-optimal design generators.

Original languageEnglish (US)
Title of host publicationProceedings of the Fifth International Workshop on Software and Performance, WOSP'05
Pages131-136
Number of pages6
StatePublished - 2005
Event5th International Workshop on Software and Performance, WOSP'05 - Illes Balears, Spain
Duration: Jul 12 2005Jul 14 2005

Other

Other5th International Workshop on Software and Performance, WOSP'05
CountrySpain
CityIlles Balears
Period7/12/057/14/05

Fingerprint

Screening
Testing
Experiments
Software testing
Design of experiments
Optimal design

Keywords

  • Covering Arrays
  • D-optimal Designs
  • Performance Testing

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Hoskins, D. S., Colbourn, C., & Montgomery, D. (2005). Software performance testing usina covering arrays: Efficient screening designs with categorical factors. In Proceedings of the Fifth International Workshop on Software and Performance, WOSP'05 (pp. 131-136)

Software performance testing usina covering arrays : Efficient screening designs with categorical factors. / Hoskins, Dean S.; Colbourn, Charles; Montgomery, Douglas.

Proceedings of the Fifth International Workshop on Software and Performance, WOSP'05. 2005. p. 131-136.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Hoskins, DS, Colbourn, C & Montgomery, D 2005, Software performance testing usina covering arrays: Efficient screening designs with categorical factors. in Proceedings of the Fifth International Workshop on Software and Performance, WOSP'05. pp. 131-136, 5th International Workshop on Software and Performance, WOSP'05, Illes Balears, Spain, 7/12/05.
Hoskins DS, Colbourn C, Montgomery D. Software performance testing usina covering arrays: Efficient screening designs with categorical factors. In Proceedings of the Fifth International Workshop on Software and Performance, WOSP'05. 2005. p. 131-136
Hoskins, Dean S. ; Colbourn, Charles ; Montgomery, Douglas. / Software performance testing usina covering arrays : Efficient screening designs with categorical factors. Proceedings of the Fifth International Workshop on Software and Performance, WOSP'05. 2005. pp. 131-136
@inproceedings{ba7dd12f12714ff8bedc1f824ed8f860,
title = "Software performance testing usina covering arrays: Efficient screening designs with categorical factors",
abstract = "Classical Design of Experiment (DOE) techniques have been in use for many years to aid in the performance testing of systems. In particular fractional factorial designs have been used in cases with many numerical factors to reduce the number of experimental runs necessary. For experiments involving categorical factors, this is not the case; experimenters regularly resort to exhaustive (full factorial) experiments. Recently, D-optimal designs have been used to reduce numbers of tests for experiments involving categorical factors because of their flexibility, but not necessarily because they can closely approximate full factorial results. In commonly used statistical packages, the only generic alternative for reduced experiments involving categorical factors is afforded by optimal designs. The extent to which D-optimal designs succeed in estimating exhaustive results has not been evaluated, and it is natural to determine this. An alternative design based on covering arrays may offer a better approximation of full factorial data. Covering arrays are used in software testing for accurate coverage of interactions, while D-optimal and factorial designs measure the amount of interaction. Initial work involved exhaustive generation of designs in order to compare covering arrays and D-optimal designs in approximating full factorial designs. In that setting, covering arrays provided better approximations of full factorial analysis.while ensuring coverage of all small interactions. Here we examine commercially viable covering array and D-optimal design generators to compare designs. Commercial covering array generators, while not as good as exhaustively generated designs, remain competitive with D-optimal design generators.",
keywords = "Covering Arrays, D-optimal Designs, Performance Testing",
author = "Hoskins, {Dean S.} and Charles Colbourn and Douglas Montgomery",
year = "2005",
language = "English (US)",
isbn = "1595930876",
pages = "131--136",
booktitle = "Proceedings of the Fifth International Workshop on Software and Performance, WOSP'05",

}

TY - GEN

T1 - Software performance testing usina covering arrays

T2 - Efficient screening designs with categorical factors

AU - Hoskins, Dean S.

AU - Colbourn, Charles

AU - Montgomery, Douglas

PY - 2005

Y1 - 2005

N2 - Classical Design of Experiment (DOE) techniques have been in use for many years to aid in the performance testing of systems. In particular fractional factorial designs have been used in cases with many numerical factors to reduce the number of experimental runs necessary. For experiments involving categorical factors, this is not the case; experimenters regularly resort to exhaustive (full factorial) experiments. Recently, D-optimal designs have been used to reduce numbers of tests for experiments involving categorical factors because of their flexibility, but not necessarily because they can closely approximate full factorial results. In commonly used statistical packages, the only generic alternative for reduced experiments involving categorical factors is afforded by optimal designs. The extent to which D-optimal designs succeed in estimating exhaustive results has not been evaluated, and it is natural to determine this. An alternative design based on covering arrays may offer a better approximation of full factorial data. Covering arrays are used in software testing for accurate coverage of interactions, while D-optimal and factorial designs measure the amount of interaction. Initial work involved exhaustive generation of designs in order to compare covering arrays and D-optimal designs in approximating full factorial designs. In that setting, covering arrays provided better approximations of full factorial analysis.while ensuring coverage of all small interactions. Here we examine commercially viable covering array and D-optimal design generators to compare designs. Commercial covering array generators, while not as good as exhaustively generated designs, remain competitive with D-optimal design generators.

AB - Classical Design of Experiment (DOE) techniques have been in use for many years to aid in the performance testing of systems. In particular fractional factorial designs have been used in cases with many numerical factors to reduce the number of experimental runs necessary. For experiments involving categorical factors, this is not the case; experimenters regularly resort to exhaustive (full factorial) experiments. Recently, D-optimal designs have been used to reduce numbers of tests for experiments involving categorical factors because of their flexibility, but not necessarily because they can closely approximate full factorial results. In commonly used statistical packages, the only generic alternative for reduced experiments involving categorical factors is afforded by optimal designs. The extent to which D-optimal designs succeed in estimating exhaustive results has not been evaluated, and it is natural to determine this. An alternative design based on covering arrays may offer a better approximation of full factorial data. Covering arrays are used in software testing for accurate coverage of interactions, while D-optimal and factorial designs measure the amount of interaction. Initial work involved exhaustive generation of designs in order to compare covering arrays and D-optimal designs in approximating full factorial designs. In that setting, covering arrays provided better approximations of full factorial analysis.while ensuring coverage of all small interactions. Here we examine commercially viable covering array and D-optimal design generators to compare designs. Commercial covering array generators, while not as good as exhaustively generated designs, remain competitive with D-optimal design generators.

KW - Covering Arrays

KW - D-optimal Designs

KW - Performance Testing

UR - http://www.scopus.com/inward/record.url?scp=33749000323&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33749000323&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:33749000323

SN - 1595930876

SN - 9781595930873

SP - 131

EP - 136

BT - Proceedings of the Fifth International Workshop on Software and Performance, WOSP'05

ER -