Software performance testing usina covering arrays: Efficient screening designs with categorical factors

Research output: Chapter in Book/Report/Conference proceedingConference contribution

17 Scopus citations

Abstract

Classical Design of Experiment (DOE) techniques have been in use for many years to aid in the performance testing of systems. In particular fractional factorial designs have been used in cases with many numerical factors to reduce the number of experimental runs necessary. For experiments involving categorical factors, this is not the case; experimenters regularly resort to exhaustive (full factorial) experiments. Recently, D-optimal designs have been used to reduce numbers of tests for experiments involving categorical factors because of their flexibility, but not necessarily because they can closely approximate full factorial results. In commonly used statistical packages, the only generic alternative for reduced experiments involving categorical factors is afforded by optimal designs. The extent to which D-optimal designs succeed in estimating exhaustive results has not been evaluated, and it is natural to determine this. An alternative design based on covering arrays may offer a better approximation of full factorial data. Covering arrays are used in software testing for accurate coverage of interactions, while D-optimal and factorial designs measure the amount of interaction. Initial work involved exhaustive generation of designs in order to compare covering arrays and D-optimal designs in approximating full factorial designs. In that setting, covering arrays provided better approximations of full factorial analysis.while ensuring coverage of all small interactions. Here we examine commercially viable covering array and D-optimal design generators to compare designs. Commercial covering array generators, while not as good as exhaustively generated designs, remain competitive with D-optimal design generators.

Original languageEnglish (US)
Title of host publicationProceedings of the Fifth International Workshop on Software and Performance, WOSP'05
Pages131-136
Number of pages6
StatePublished - Dec 1 2005
Event5th International Workshop on Software and Performance, WOSP'05 - Illes Balears, Spain
Duration: Jul 12 2005Jul 14 2005

Publication series

NameProceedings of the Fifth International Workshop on Software and Performance, WOSP'05

Conference

Conference5th International Workshop on Software and Performance, WOSP'05
CountrySpain
CityIlles Balears
Period7/12/057/14/05

Keywords

  • Covering Arrays
  • D-optimal Designs
  • Performance Testing

ASJC Scopus subject areas

  • Engineering(all)

Fingerprint Dive into the research topics of 'Software performance testing usina covering arrays: Efficient screening designs with categorical factors'. Together they form a unique fingerprint.

  • Cite this

    Hoskins, D. S., Colbourn, C., & Montgomery, D. (2005). Software performance testing usina covering arrays: Efficient screening designs with categorical factors. In Proceedings of the Fifth International Workshop on Software and Performance, WOSP'05 (pp. 131-136). (Proceedings of the Fifth International Workshop on Software and Performance, WOSP'05).