Real-World Use Cases

Combinatorial Testing is a discipline that can be applied to a wide variety of fields. In the following we list some use cases where Combinatorial Testing (and in particular CTLog) can be successfully applied.

Finding bugs in software: The GCC compiler

Consider a program that receives a set of input parameters, whose values will affect the execution of the program. We would like to capture as many errors as possible for the different combinations of parameters’ values accepted by the program. Combinatorial Testing techniques can be applied in this case to try to test as many parameters’ interactions as possible using a reasonable amount of test cases.

As an example, consider the GCC compiler, which is an optimizing compiler produced by the GNU Project supporting various programming languages, hardware architectures and operating systems. GCC has 189 input parameters of domain 2 and 10 input parameters of domain 3, as well as 40 user constraints that rule out forbidden combinations of the parameters. Checking all the parameter space would require \(4.6 \cdot 10^{61}\) tests (settings to the input parameters) on the particular selected scenario (set of benchmarks where GCC is applied). Even if the test would only take one second when applied to the scenario, we would need the age of the universe to complete all the possible tests according to the parameter space of GCC.

However, we could reduce the number of required test scenarios by checking only the interactions of just 2 parameters. If we apply a naive enumeration approach, where at each test we only track one single interaction, we would need \(82809\) tests. However, by using CT techniques we can reduce this number to just \(15\) tests. If all these 15 tests do not produce any error we know that there is no interaction of two parameters of GCC that causes a failure or bug in our selected scenario.

Testing Autonomous Driving Systems

Autonomous Driving Systems are becoming more present in our daily lives. In recet years, we have observed an increase on the number of car manufacturers that offer some kind of Autonomous Driving technology for some of their cars. It is clear that these kind of systems are critical in terms of their safety and, unfortunately, we have already encountered fatal accidents involving Autonomous Driving Systems. Therefore, testing these systems to capture errors before they appear in real-world scenarios is an essential task, especially when a failure can involve human losses.

Combinatorial Testing is a relevant tool that can help on testing Autonomous Driving Systems. There is an active community focussed on testing and debugging Autonomous Driving Systems, being Combinatorial Testing some of the techniques that they apply. In [LCS+21], authors apply Combinatorial Testing to generate different scenarios for an Autonomous Driving System. CTLog could be easily integrated into these kind of research to generate test suites, as researchers would have access to a diverse variety of CT algorithms.