Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures

Changhai Nie, Huayao Wu, Xintao Niu, Fei Ching Kuo, Hareton Leung, Charles Colbourn

Research output: Contribution to journalArticle

24 Citations (Scopus)

Abstract

Context: Software behavior depends on many factors, and some failures occur only when certain factors interact. This is known as an interaction triggered failure, and the corresponding selection of factor values can be modeled as a Minimal Failure-causing Schema (MFS). (An MFS involving m factors is an m-MFS.) Combinatorial Testing (CT) has been developed to exercise ("hit") all MFS with few tests. Adaptive Random Resting (ART) endeavors to make tests as different as possible, ensuring that testing of MFS is not unnecessarily repeated. Random Testing (RT) chooses tests at random without regard to the MFS already treated. CT might be expected to improve on RT for finding interaction triggered faults, and yet some studies report no significant difference. CT can also be expected to be better than ART, and yet other studies report that ART can be much better than RT. In light of these, the relative merits of CT, ART, and RT for finding interaction triggered faults are unclear. Objective: To investigate the relationships among CT, ART, and RT, we conduct the first complete and systematic comparison for the purpose of hitting MFS. Method: A systematic review of six aspects of CT, RT and ART is conducted first. Then two kinds of experiments are used to compare them under four metrics. Results: ART improves upon RT, but t-way CT is better than both. In hitting t'-MFS the advantage is typically in the range from 10% to 30% when t = t', but becomes much smaller when t' < t, and there may be no advantage when t' > t. The latter case may explain the studies reporting no significant difference between RT and CT. Conclusion: RT is easily implemented. However, depending on its implementation, ART can improve upon RT. CT does as well as ART whether or nott' = t, but provides a valuable improvement in the cases when t' = t.

Original languageEnglish (US)
Pages (from-to)198-213
Number of pages16
JournalInformation and Software Technology
Volume62
Issue number1
DOIs
StatePublished - 2015

Fingerprint

Testing

Keywords

  • Adaptive Random Testing (ART)
  • Combinatorial Testing (CT)
  • Interaction Triggered Failure (ITF)
  • Minimal Failure-causing Schema (MFS)
  • Software testing Random Testing (RT)

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Computer Science Applications

Cite this

Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures. / Nie, Changhai; Wu, Huayao; Niu, Xintao; Kuo, Fei Ching; Leung, Hareton; Colbourn, Charles.

In: Information and Software Technology, Vol. 62, No. 1, 2015, p. 198-213.

Research output: Contribution to journalArticle

Nie, Changhai ; Wu, Huayao ; Niu, Xintao ; Kuo, Fei Ching ; Leung, Hareton ; Colbourn, Charles. / Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures. In: Information and Software Technology. 2015 ; Vol. 62, No. 1. pp. 198-213.
@article{417d72f04e1e4247b67193912ce5e47e,
title = "Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures",
abstract = "Context: Software behavior depends on many factors, and some failures occur only when certain factors interact. This is known as an interaction triggered failure, and the corresponding selection of factor values can be modeled as a Minimal Failure-causing Schema (MFS). (An MFS involving m factors is an m-MFS.) Combinatorial Testing (CT) has been developed to exercise ({"}hit{"}) all MFS with few tests. Adaptive Random Resting (ART) endeavors to make tests as different as possible, ensuring that testing of MFS is not unnecessarily repeated. Random Testing (RT) chooses tests at random without regard to the MFS already treated. CT might be expected to improve on RT for finding interaction triggered faults, and yet some studies report no significant difference. CT can also be expected to be better than ART, and yet other studies report that ART can be much better than RT. In light of these, the relative merits of CT, ART, and RT for finding interaction triggered faults are unclear. Objective: To investigate the relationships among CT, ART, and RT, we conduct the first complete and systematic comparison for the purpose of hitting MFS. Method: A systematic review of six aspects of CT, RT and ART is conducted first. Then two kinds of experiments are used to compare them under four metrics. Results: ART improves upon RT, but t-way CT is better than both. In hitting t'-MFS the advantage is typically in the range from 10{\%} to 30{\%} when t = t', but becomes much smaller when t' < t, and there may be no advantage when t' > t. The latter case may explain the studies reporting no significant difference between RT and CT. Conclusion: RT is easily implemented. However, depending on its implementation, ART can improve upon RT. CT does as well as ART whether or nott' = t, but provides a valuable improvement in the cases when t' = t.",
keywords = "Adaptive Random Testing (ART), Combinatorial Testing (CT), Interaction Triggered Failure (ITF), Minimal Failure-causing Schema (MFS), Software testing Random Testing (RT)",
author = "Changhai Nie and Huayao Wu and Xintao Niu and Kuo, {Fei Ching} and Hareton Leung and Charles Colbourn",
year = "2015",
doi = "10.1016/j.infsof.2015.02.008",
language = "English (US)",
volume = "62",
pages = "198--213",
journal = "Information and Software Technology",
issn = "0950-5849",
publisher = "Elsevier",
number = "1",

}

TY - JOUR

T1 - Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures

AU - Nie, Changhai

AU - Wu, Huayao

AU - Niu, Xintao

AU - Kuo, Fei Ching

AU - Leung, Hareton

AU - Colbourn, Charles

PY - 2015

Y1 - 2015

N2 - Context: Software behavior depends on many factors, and some failures occur only when certain factors interact. This is known as an interaction triggered failure, and the corresponding selection of factor values can be modeled as a Minimal Failure-causing Schema (MFS). (An MFS involving m factors is an m-MFS.) Combinatorial Testing (CT) has been developed to exercise ("hit") all MFS with few tests. Adaptive Random Resting (ART) endeavors to make tests as different as possible, ensuring that testing of MFS is not unnecessarily repeated. Random Testing (RT) chooses tests at random without regard to the MFS already treated. CT might be expected to improve on RT for finding interaction triggered faults, and yet some studies report no significant difference. CT can also be expected to be better than ART, and yet other studies report that ART can be much better than RT. In light of these, the relative merits of CT, ART, and RT for finding interaction triggered faults are unclear. Objective: To investigate the relationships among CT, ART, and RT, we conduct the first complete and systematic comparison for the purpose of hitting MFS. Method: A systematic review of six aspects of CT, RT and ART is conducted first. Then two kinds of experiments are used to compare them under four metrics. Results: ART improves upon RT, but t-way CT is better than both. In hitting t'-MFS the advantage is typically in the range from 10% to 30% when t = t', but becomes much smaller when t' < t, and there may be no advantage when t' > t. The latter case may explain the studies reporting no significant difference between RT and CT. Conclusion: RT is easily implemented. However, depending on its implementation, ART can improve upon RT. CT does as well as ART whether or nott' = t, but provides a valuable improvement in the cases when t' = t.

AB - Context: Software behavior depends on many factors, and some failures occur only when certain factors interact. This is known as an interaction triggered failure, and the corresponding selection of factor values can be modeled as a Minimal Failure-causing Schema (MFS). (An MFS involving m factors is an m-MFS.) Combinatorial Testing (CT) has been developed to exercise ("hit") all MFS with few tests. Adaptive Random Resting (ART) endeavors to make tests as different as possible, ensuring that testing of MFS is not unnecessarily repeated. Random Testing (RT) chooses tests at random without regard to the MFS already treated. CT might be expected to improve on RT for finding interaction triggered faults, and yet some studies report no significant difference. CT can also be expected to be better than ART, and yet other studies report that ART can be much better than RT. In light of these, the relative merits of CT, ART, and RT for finding interaction triggered faults are unclear. Objective: To investigate the relationships among CT, ART, and RT, we conduct the first complete and systematic comparison for the purpose of hitting MFS. Method: A systematic review of six aspects of CT, RT and ART is conducted first. Then two kinds of experiments are used to compare them under four metrics. Results: ART improves upon RT, but t-way CT is better than both. In hitting t'-MFS the advantage is typically in the range from 10% to 30% when t = t', but becomes much smaller when t' < t, and there may be no advantage when t' > t. The latter case may explain the studies reporting no significant difference between RT and CT. Conclusion: RT is easily implemented. However, depending on its implementation, ART can improve upon RT. CT does as well as ART whether or nott' = t, but provides a valuable improvement in the cases when t' = t.

KW - Adaptive Random Testing (ART)

KW - Combinatorial Testing (CT)

KW - Interaction Triggered Failure (ITF)

KW - Minimal Failure-causing Schema (MFS)

KW - Software testing Random Testing (RT)

UR - http://www.scopus.com/inward/record.url?scp=84932626039&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84932626039&partnerID=8YFLogxK

U2 - 10.1016/j.infsof.2015.02.008

DO - 10.1016/j.infsof.2015.02.008

M3 - Article

VL - 62

SP - 198

EP - 213

JO - Information and Software Technology

JF - Information and Software Technology

SN - 0950-5849

IS - 1

ER -